My Notes
Table of Contents
- 1. PHP
- 1.1. Installation
- 1.2. Configuration & CLI
- 1.2.1. CLI
- 1.2.2. Basics
- 1.2.3.
[PHP]- 1.2.3.1. magic quotes
- 1.2.3.2. allow_url_fopen
- 1.2.3.3. auto_prepend_file
- 1.2.3.4. expose_php
- 1.2.3.5. session
- 1.2.3.6. extension_dir php.ini:extension_dir
- 1.2.3.7. max_execution_time php.ini:max_execution_time
- 1.2.3.8. include_path
- 1.2.3.9. short_open_tag
- 1.2.3.10. memory_limit default 128M PHP_INI_ALL
- 1.2.4. Enable extension php.ini:extension_dir
- 1.2.5. prod
- 1.2.6. dev
- 1.2.7. Mail
- 1.2.8. Error log
- 1.2.9. Trigger error
- 1.2.10. Custom error/warning/notice hanlder
- 1.2.11. register_shutdown_function
- 1.2.12. opcache
- 1.3. Kill
- 1.4. Constant
definevsconst - 1.5. Protocols
- 1.6.
$_SERVER - 1.7.
$_FILESPOST method uploads - 1.8.
$_COOKIEvalues may also exist in$_REQUEST - 1.9. Filesystem
- 1.10. Module, Extension
- 1.11. Locale
- 1.12. Date
- 1.13. Bitwise Operator
- 1.14. Ternary Operator, Null Coalescing Operator, Elvis Operator, Short
if - 1.15. Short Echo
- 1.16. Error Control Operator
- 1.17. Spaceship Operator
- 1.18. Multibyte php:multibyte
- 1.19. Array
- 1.19.1. Spread Operator / Splat
- 1.19.2. list, compact, extract
- 1.19.3.
array_merge, array_merge_recursive php:array_merge - 1.19.4. Append an array or an element to another array
- 1.19.5. Union
$array1 + $array2 - 1.19.6. Last Element, First Element, First Key
- 1.19.7.
array_shift,array_unshiftprepend elements,array_pop - 1.19.8.
in_array - 1.19.9. Search by a value and return keys
- 1.19.10. Search by keys and return values in order
- 1.19.11. Search by value and return the first key by value
array_search - 1.19.12. Delete by a value
- 1.19.13. Change Case for All Keys
- 1.19.14. Change keys only
- 1.19.15. Get values of a column in a multidimensional array
- 1.19.16.
array_diff,array_diff_key,array_diff_assoc,array_intersect,array_intersect_assoc - 1.19.17.
array_key_exists - 1.19.18.
array_map - 1.19.19.
array_spliceInsert to any position of an array - 1.19.20.
array_sliceExtract a slice of the array - 1.19.21.
array_fillfill values - 1.19.22.
array_fill_keys - 1.19.23.
array_walk - 1.19.24.
array_filter - 1.19.25.
array_reduce - 1.19.26.
array_sum() - 1.19.27. Sort arrays
- 1.19.28.
array_reverse - 1.19.29. array_keys, array_values
- 1.19.30. array_unique
- 1.19.31. array_flip
- 1.19.32. range
- 1.19.33. Associative array
- 1.19.34. Set array value by deep path
- 1.19.35. If it's numeric and continuous array
- 1.19.36. Custom pretty print array
- 1.20. String
- 1.20.1. Conversion to string
- 1.20.2. Empty string
- 1.20.3. Multibyte - refer to php:multibyte
- 1.20.4. String to HTML,
- 1.20.5. Regex
- 1.20.6. Read string line by line
- 1.20.7. String Ellipsis, Trim php:ellipsis
- 1.20.8. Encode HTML
- 1.20.9. URL Encode php:urlencode
- 1.20.10. strip_tags
- 1.20.11. UTF-8 encoding
- 1.20.12. JSON
- 1.20.13. Back slashes for quotes and other characters
- 1.20.14. Replace php:str_replace
- 1.20.15. Replace first or last occurance
- 1.20.16. Find the position of the first/last occurrence of a substring in a string
- 1.20.17. First character uppercase/lowercase -
ucfirst()lcfirst() - 1.20.18. First letter of each word uppercase
ucwords() - 1.20.19. Start with / End with
- 1.20.20. sprintf and printf
- 1.20.21. strtr
- 1.20.22. Random base64 string
- 1.20.23. Hash: md5, sha1
- 1.20.24. Password Hash
- 1.20.25. encrypt, decrypt
- 1.20.26. Crypt
- 1.20.27. Random
- 1.20.28. True or False on string
- 1.20.29. If string is an integer
- 1.21. Conversion to boolean
- 1.22. URL
- 1.23. Random Integer Number
- 1.24. HTML encode and decode
- 1.25. Heredoc, Nowdoc
- 1.26. Buffer
- 1.27. Template System
- 1.28. Image
- 1.29. Apply XSLT on XML
- 1.30. Block Referral Traffic
- 1.31. PHPDoc
- 1.32. Database
- 1.33. Function
- 1.34. OOP
- 1.34.1. Sample
- 1.34.2. Instantiate Dynamically
- 1.34.3. OOP Features
- 1.34.4. Inheritance
- 1.34.5. Final keyword
- 1.34.6. Abstract Class
- 1.34.7. Interface
- 1.34.8. Interface vs Abstract Class
- 1.34.9. Trait
- 1.34.10. Magic Methods
- 1.34.11. Magic Constants php:magic constants
- 1.34.12. is_object, is_a, is_subclass_of, get_class(), get_parent_class(), get_called_class()
- 1.34.13. Clone, copy and reference
- 1.34.14. Autoload Classes
- 1.34.15. Design Pattern
- 1.34.15.1. Singleton (Creational DP)
- 1.34.15.2. Prototype (Creational DP)
- 1.34.15.3. Builder (Creational DP)
- 1.34.15.4. Lazy Initialization
- 1.34.15.5. Factory Method (Creational DP)
- 1.34.15.6. Facade (Structural DP)
- 1.34.15.7. Proxy (Structural DP)
- 1.34.15.8. Dependency Injection
- 1.34.15.9. Decorator (Structural DP)
- 1.34.15.10. Adapter (Structural DP)
- 1.34.15.11. Strategy Pattern (Behavioral DP)
- 1.34.15.12. Iterator (Behavioral DP)
- 1.34.15.13. Observer Pattern (Behavioral DP)
- 1.34.15.14. Chain of Responsibility (Behavioral DP) - Composition/Aggregation + Dependency Injection
- 1.34.16. Architectural Pattern
- 1.34.17. Clean Code
- 1.34.18. Namespace
- 1.34.19. Cannot redeclare class
- 1.34.20. ReflectionClass, get_class_vars(), get_object_vars()
- 1.34.21. Add capabilities to a class without making it bigger
- 1.34.22. Cast to object
- 1.35. Global
- 1.36. Http Request - cURL, file_get_contents
- 1.37. HTTP Header
- 1.38. Throwable Interface
- 1.39. Exception
- 1.40. Error Exception
- 1.41. Type declaration
- 1.42. Pass by Reference
- 1.43. Troubleshooting
- 1.44. Naming Convention
- 1.45. Shell Exec Command
- 1.46. Session
- 1.47. Predefined Interfaces and Classes
- 1.48. Composer
- 1.49. PHPUnit
- 1.50. Standard PHP Library (SPL)
- 1.51. Libraries
- 1.52. Run PHP Online in multiple versions
- 1.53. Coding Standard
- 2. Drupal
- 2.1. Install Drupal on Windows
- 2.2. Update Core
- 2.3. Update Module
- 2.4. Module
- 2.4.1. Core Modules
- 2.4.2. Custom Module
- 2.4.3. sqlsrv
- 2.4.4. Pathauto d7:proj:pathauto
- 2.4.5. Feeds - Package: Feeds
- 2.4.6. Feeds Tamper - feeds_tamper - Package: Feeds
- 2.4.7. Feeds Tamper Importer - feeds_tamper_importer - Package: Feeds
- 2.4.8. Feeds XPath Parser - feeds_xpathparser - Package: Feeds
- 2.4.9. Feeds entity processor - feeds_entity_processor - Package: Feeds
- 2.4.10. XML Sitemap
- 2.4.11. ThemeKey d7:module:themekey
- 2.4.12. Devel
- 2.4.13. Administrator menu - admin_menu - Package: Administration
- 2.4.14. Administration Views - admin_views - Package: Administration
- 2.4.15. Administer Users by Role - administerusersbyrole
- 2.4.16. Variable
- 2.4.17. Libraries
- 2.4.18. Module Filter
- 2.4.19. Geocoder
- 2.4.20. Entity View Modes - entity_view_mode
- 2.4.21. wysiwyg
- 2.4.22. IMCE Wysiwyg bridge: imce_wysiwyg
- 2.4.23. IMCE
- 2.4.24. ckeditor
- 2.4.25. nodeblock
- 2.4.26. Block Group
- 2.4.27. MultiBlock d7:multiblock
- 2.4.28. Conditional Fields
- 2.4.29. Search404
- 2.4.30. Google Analytics
- 2.4.31. Elysia Cron
- 2.4.32. SMTP Authentication Support
- 2.4.33. Chaos Tools - ctools
- 2.4.34. Address Field - addressfield - Package: Fields
- 2.4.35. Email - email - Package: Fields
- 2.4.36. Geofield - Package: Fields
- 2.4.37. Viewfield - Package: Fields
- 2.4.38. Field collection - Package: Fields d7:proj:field_collection
- 2.4.39. Social Field - socialfield - Package: Fields
- 2.4.40. Video Embed Field - video_embed_field - Package: Media
- 2.4.41. Display Suite - ds - Package: Display Suite
- 2.4.42. Meta Tag - metatag
- 2.4.43. Entity Reference - entityreference
- 2.4.44. File Entity
- 2.4.45. Webform d7:webform
- 2.4.46. Webform Validation
- 2.4.47. Panels
- 2.4.48. Facet API d7:facetapi
- 2.4.49. Search API - search_api
- 2.4.49.1. Glossary
- 2.4.49.2. Index
- 2.4.49.3. Create a search view or a search page
- 2.4.49.4. Convert an existing view to search api
- 2.4.49.5. Configure Facet, Facet Blocks
- 2.4.49.6. Views Filter > Search: Fulltext search
- 2.4.49.7. Current Search Block
- 2.4.49.8. Other useful modules
- 2.4.49.9. Developer Documentation
- 2.4.49.10. API of Search API
- 2.4.50. Search API Solr - search_api_solr - Package: Search
- 2.4.51. Search API Solr Overrides - search_api_solr_overrides
- 2.4.52. Views
- 2.4.53. Better Exposed Filters
- 2.4.54. Views Bulk Operations
- 2.4.55. Administration views
- 2.4.56. Media
- 2.4.57. Internationalization - i18n
- 2.4.58. S3 File System - s3fs d7:m:s3fs
- 2.4.59. Password Policy - password_policy
- 2.4.60. Secure Login - securelogin
- 2.4.61. Secure Review - security_review
- 2.4.62. Security Kit - seckit
- 2.5. Debug
- 2.6. Configuration, Setting, Maintenance Mode
- 2.7. URL Alias
- 2.8. Link, Module Path, Theme Path, File Path
- 2.9. Translate
- 2.10. Global Variables
- 2.11. Common Functions d7:functions
- 2.12. Theme API
- 2.12.1. render() vs drupal_render()
- 2.12.2. theme()
- 2.12.3. System Theme
- 2.12.3.1. Fast detect if it's a node page
- 2.12.3.2. Include file in *.tpl.php
- 2.12.3.3. .info (required)
- 2.12.3.4. html.tpl.php
- 2.12.3.5. page.tpl.php
- 2.12.3.6. region.tpl.php
- 2.12.3.7. block.tpl.php
- 2.12.3.8. node.tpl.php d7:template:node
- 2.12.3.9. field.tpl.php
- 2.12.3.10. comment-wrapper.tpl.php
- 2.12.3.11. comment.tpl.php
- 2.12.3.12. template.php
- 2.12.3.13. logo.png
- 2.12.3.14. screenshot.png
- 2.12.4. Form Theme
- 2.12.5. Region
- 2.12.6. Block
- 2.12.7. Custom Theme Function
- 2.12.8. Sub theme
- 2.12.9. 404 page
- 2.12.10. Contributed themes
- 2.12.11. Render a field of a node, a term
- 2.13. Drush
- 2.14. Node API d7:api:node
- 2.15. Views d7:viewsapi
- 2.15.1. UI
- 2.15.2. Entity Reference View
- 2.15.3. View Object
- 2.15.4. hook_views_api
- 2.15.5. hook_views_query_alter
- 2.15.6. hook_views_pre_build
- 2.15.7. hook_views_pre_render
- 2.15.8. Custom View Filter hook_views_data_alter
- 2.15.9. Better Exposed Filters Module d7:better exposed filters
- 2.15.10. Custom Views Plugin hook_views_plugins
- 2.15.11. Views API Order
- 2.15.12. Embed View and Display
- 2.15.13. View Theming d7:View_Theming
- 2.16. Ellipsis d7:ellipsis
- 2.17. Form
- 2.18. Action
- 2.19. Field API d7:api:field
- 2.20. Database d7:database
- 2.20.1. Close Comment for Existing Nodes
- 2.20.2. Structure
- 2.20.2.1. node
- 2.20.2.2. field_config
- 2.20.2.3. field_data_body
- 2.20.2.4.
field_data_[field_my_text_or_longtext] - 2.20.2.5.
field_data_[field_my_date_unix_timestamp] - 2.20.2.6.
field_data_[field_custom_term_reference] - 2.20.2.7.
field_data_field_mileage_unitslist_text - 2.20.2.8.
field_data_field_contributorentity reference - 2.20.2.9. taxonomy_term_data
- 2.20.2.10. taxonomy_vocabular
- 2.20.2.11. url_alias
- 2.20.2.12. Export Query
- 2.20.2.13. Export File URI, file_managed d7:db:file uri
- 2.20.2.14. file_usage
- 2.20.2.15. system
- 2.20.2.16. users
- 2.20.2.17. users_role
- 2.20.2.18. role
- 2.21. Query
- 2.22. Entity
- 2.23. Entityform
- 2.24. User
- 2.25. Menu d7:menu
- 2.26. Taxonomy
- 2.26.1. Get taxonomy terms of a node, ready for display with language
- 2.26.2. Get multiple terms by id, taxonomy_term_load_multiple
- 2.26.3. Get term by name, taxonomy_get_term_by_name
- 2.26.4. Get all children of a term, taxonomy_get_children
- 2.26.5. Get a vocabulary by name and all terms
- 2.26.6. Get terms of a vocabulary ready for form
- 2.27. Cache
- 2.28. File
- 2.29. File Permissions d7:file permissions
- 2.30. Image Handling (Core)
- 2.31. Basic Ajax
- 2.32. Javascript API
- 2.33. Email
- 2.34. Life Cycle
- 2.35. Hook System d7:hook system
- 2.36. OOP
- 3. WordPress
- 3.1. Core Update
- 3.2. Migrate database
- 3.3. Redirect a path a Wordpress website wp:redirect path to wp
- 3.4. Load WP Core in sub directory under same domain
- 3.5. Global Variables
- 3.5.1.
$wp_version - 3.5.2.
$_GET,$_POST,$_COOKIE,$_SERVER - 3.5.3.
$wp_query$GLOBALS['wp_query']wp:global:wp_query - 3.5.4.
$wp_rewritewp:global:rewrite - 3.5.5.
$postwp:global:post - 3.5.6.
$wpdbwp:global:wpdb - 3.5.7.
$current_userwp:global:current_user - 3.5.8.
$wp_filter,$wp_current_filterwp:global:wp_filter wp:global:wp_current_filter - 3.5.9.
$templatewp:global:template - 3.5.10. Image Sizes
$_wp_additional_image_sizes - 3.5.11. Post statuses
$wp_post_statuseswp:global:wp_post_statuses - 3.5.12.
$wp_customizewp:global:wp_customize
- 3.5.1.
- 3.6. wp-config.php
- 3.7. Options, Settings wp:options
- 3.8. Email PHPMailer
- 3.9. PHP redirect wp:php:redirect
- 3.10. WP-CLI wp-cli
- 3.10.1. WP CLI Install, Upgrade
- 3.10.2. Global Parameters
- 3.10.3. Command documentation
- 3.10.4. user
- 3.10.5. core
- 3.10.6. plugin
- 3.10.7. package
- 3.10.8. theme
- 3.10.9. login
- 3.10.10. media
- 3.10.11. post wp-cli:post
- 3.10.12. db check
- 3.10.13. db search
- 3.10.14. db export
- 3.10.15. db size
- 3.10.16. db query
- 3.10.17. search-replace wp-cli:search-replace
- 3.10.18. taxonomy
- 3.10.19. term
- 3.10.20. eval
- 3.10.21. config list
- 3.10.22. embed wp-cli:embed
- 3.10.23. Common Issues
- 3.11. URL, Current URL, and slug wp:current url
- 3.12. WP_Post Post Object wp:post
- 3.13.
WP_User, user object wp:user - 3.14. WP Custom Fields wp:Custom_Fields
- 3.15. WP_Query The Loop wp:The_Loop wp:query:main
- 3.15.1. Syntax
- 3.15.2. WP_Query
- 3.15.2.1. Basics
- 3.15.2.2. Get child posts
- 3.15.2.3. Get attached images
- 3.15.2.4. Loop the main loop
- 3.15.2.5. Loop with
query_posts()wp:f:query_posts - 3.15.2.6. Loop with
new WP_Query() - 3.15.2.7. Loop with
get_posts() - 3.15.2.8. Loop outside a WordPress Loop wp:loop:outside
- 3.15.2.9. Loop inside loop, Loop Stack wp:loop:stack
- 3.15.2.10.
is_*()methods wp:wp_query:m:is_* - 3.15.2.11.
found_posts - 3.15.2.12.
post_count - 3.15.2.13.
current_post - 3.15.2.14.
in_the_loopWhether posts are in a loop - 3.15.2.15.
postCurrent post - 3.15.2.16.
postsList of posts - 3.15.2.17.
queryQuery vars set by the user - 3.15.2.18.
query_varsQuery vars, after parsing wp:wp_query:query_vars - 3.15.2.19.
have_posts() - 3.15.2.20.
the_post() - 3.15.2.21.
query( $query )wp:wp_query:m:query - 3.15.2.22.
parse_query( $query = '' )wp:wp_query:m:parse_query - 3.15.2.23.
get_posts()wp:wp_query:m:get_posts
- 3.15.3. WP_Meta_Query
- 3.15.4. WP Tax Query
- 3.15.5. WP Order and Orderby
- 3.15.6. WP_Term_Query wp:WP_Term_Query
- 3.15.7. Review Raw Query
- 3.16. Functions
- 3.16.1. Plugin API
wp-includes/plugin.phpwp:api:plugin - 3.16.2. Transients API -
set_transientvswp_cache_setwp:cache - 3.16.3. Formatting
wp-includes/formatting.php - 3.16.4. Main API
wp-includes/functions.phpwp:api:main - 3.16.5. Translate wp:translate
- 3.16.6. User Role & Capabilities API wp:api:role-capabilities wp:roles
- 3.16.7. Query API wp:api:query
- 3.16.8. Theme functions
wp-includes/theme.php- 3.16.8.1. get_template_directory, get_template_directory_uri, get_stylesheet_directory_uri, get_stylesheet_uri
- 3.16.8.2. get_theme_mod wp:f:get_theme_mod
- 3.16.8.3. add_theme_support, remove_theme_support, current_theme_supports wp:f:add_theme_support
- 3.16.8.4.
add_editor_style( array|string $stylesheet = 'editor-style.css' ) - 3.16.8.5. get_theme_root, get_theme_root_uri
- 3.16.9. Template Tags wp:template tags
- 3.16.9.1. post-thumbnail-template.php wp:t:thumbnail
- 3.16.9.2. post-template.php wp:t:post
- 3.16.9.3. category-template.php wp:t:category
- 3.16.9.4. general-template wp:t:general
- 3.16.9.5. author-template.php
- 3.16.9.6. bookmark_template.php
- 3.16.9.7. comment-template.php
- 3.16.9.8. link-template.php
- 3.16.9.9. media-template.php
- 3.16.9.10. nav-menu-template.php
- 3.16.9.11. feed.php
- 3.16.10. Post API
wp-includes/post.phpwp:api:post- 3.16.10.1.
get_post( int|WP_Post|null $post = null, string $output = OBJECT, string $filter = 'raw' )wp:f:get_post - 3.16.10.2.
get_metadata( string $meta_type, int $object_id, string $meta_key = '', bool $single = false )wp:f:get_metadata - 3.16.10.3.
get_post_meta( int $post_id, string $key = '', bool $single = false )wp:f:get_post_meta - 3.16.10.4.
get_posts( $args = null ) : WP_Post[]|int[]wp:f:get_posts - 3.16.10.5.
wp_get_post_terms( $post_id = 0, $taxonomy = 'post_tag', $args = array() ) : WP_Term[]|WP_Errorwp:f:wp_get_post_terms - 3.16.10.6.
add_post_type_support( string $post_type, string|array $supports)wp:f:add_post_type_support - 3.16.10.7. register_post_type wp:f:register_post_type wp:add cpt
- 3.16.10.8. register_post_status wp:f:register_post_status
- 3.16.10.9. register_post_meta wp:f:register_post_meta
- 3.16.10.10. update_post_meta wp:f:update_post_meta
- 3.16.10.1.
- 3.16.11. Post Formats
wp-includes/post-formats.php - 3.16.12. Rewrite API wp:api:rewrite
- 3.16.13. Feed API
wp-includes/feed.phpwp:api:feed - 3.16.14. Dependency API
wp-includes/script-loader.phpwp:api:dependency - 3.16.15. Taxonomy API
wp-includes/taxonomy.phpwp:api:taxonomy- 3.16.15.1.
get_term_by( $field, $value, $taxonomy = '', $output = OBJECT, $filter = 'raw' ) - 3.16.15.2.
register_taxonomy( string $taxonomy, array|string $object_type, array|string $args = array() )wp:f:register_taxonomy - 3.16.15.3.
wp_update_term_count_now( array $terms, string $taxonomy )wp:f:wp_update_term_count_now - 3.16.15.4. wp:action:registered_taxonomy
- 3.16.15.5. Add a sortable column to taxonomy edit page
- 3.16.15.6.
get_taxonomies( $args = array(), $output = 'names', $operator = 'and' ) : string[]|WP_Taxonomy[]wp:f:get_taxonomies
- 3.16.15.1.
- 3.16.16. Shortcode API
wp-includes/shortcodes.phpwp:api:shortcode - 3.16.17. OEmbed API
wp-includes/embed.phpwp:api:oembed - 3.16.18. Media API
wp-includes/media.phpwp:api:media - 3.16.19. HTTP Request API
wp-includes/httpl.phpwp:api:http request - 3.16.20. Widget API
wp-includes/widgets.phpwp:api:widgets - 3.16.21. Navigation Menu Functions
wp-includes/nav-menu.php,nav-menu-template.php - 3.16.22. Toolbar API
wp-includes/admin-bar.phpwp:api:toolbar - 3.16.23. Pluggable functions
wp-includes/pluggable.phpwp:pluggable - 3.16.24. Administration API
wp-admin/includes/admin.phpwp:api:core admin- 3.16.24.1.
add_meta_box( $id, $title, $callback, $screen = null, $context = 'advanced', $priority = 'default', $callback_args = null )wp:f:add_meta_box - 3.16.24.2.
remove_meta_box( $id, $page, $context)wp:f:remove_meta_box - 3.16.24.3.
add_management_page( $page_title, $menu_title, $capability, $menu_slug, $function = '' )
- 3.16.24.1.
- 3.16.1. Plugin API
- 3.17. Default filters and actions wp:default filters
- 3.18. Filter
- 3.18.1. post_limits wp:filter:post_limits
- 3.18.2. get_meta_sql wp:filter:get_meta_sql
- 3.18.3. posts_clauses wp:filter:posts_clauses
- 3.18.4. query_vars wp:filter:query_vars
- 3.18.5. request wp:filter:request
- 3.18.6. page_link, post_link wp:filter:page_link wp:filter:post_link
- 3.18.7. url_to_postid
- 3.18.8. pre_option_(option_name) wp:filter:pre_option_*
- 3.18.9. found_posts wp:filter:found_posts
- 3.18.10. template_include wp:filter:template_include
- 3.18.11. flush_rewrite_rules_hard wp:filter:flush_rewrite_rules_hard
- 3.18.12. tiny_mce_before_init wp:filter:tiny_mce_before_init
- 3.18.13. the_title wp:filter:the_title
- 3.18.14. the_title_rss wp:filter:the_title_rss
- 3.18.15. the_content wp:filter:the_content
- 3.18.16. upload_mimes wp:filter:upload_mimes
- 3.18.17. widget_text wp:filter:widget_text
- 3.18.18. sidebars_widgets wp:filter:sidebars_widgets
- 3.18.19. show_admin_bar wp:filter:show_admin_bar
- 3.18.20. excerpt_more wp:filter:excerpt_more
- 3.18.21. excerpt_length wp:filter:excerpt_length
- 3.18.22. wp_calculate_image_srcset wp:filter:wp_calculate_image_srcset
- 3.18.23. wp_calculate_image_srcset_meta wp:filter:wp_calculate_image_srcset_meta
- 3.18.24. wp_calculate_image_sizes wp:filter:wp_calculate_image_sizes
- 3.18.25. wp_resource_hints
- 3.18.26. script_loader_src style_loader_src wp:filter:script_loader_src wp:filter:style_loader_src
- 3.18.27. imgae_size_names_choose wp:filter:image_size_names_choose
- 3.18.28. get_header_image_tag
- 3.18.29. wp_nav_menu_args wp:filter:wp_nav_menu_args
- 3.18.30. wp_get_attachment_image_src wp:filter:wp_get_attachment_image_src
- 3.18.31. wp_get_attachment_image_attributes
- 3.18.32. widget_tag_cloud_args
- 3.18.33. comments_open
- 3.18.34. deprecated_constructor_trigger_error wp:filter:deprecated_constructor_trigger_error
- 3.19. Action
- 3.19.1. activated_plugin wp:action:activated_plugin
- 3.19.2. plugins_loaded wp:action:plugins_loaded
- 3.19.3. after_setup_theme wp:action:after_setup_theme
- 3.19.4. init wp:action:init
- 3.19.5. widgets_init wp:action:widgets_init
- 3.19.6. admin_menu wp:action:admin_menu
- 3.19.7. admin_init wp:action:admin_init
- 3.19.8. parse_request wp:action:parse_request
- 3.19.9. pre_get_posts wp:action:pre_get_posts
- 3.19.10. send_headers wp:action:send_headers
- 3.19.11. wp_enqueue_scripts wp:action:wp_enqueue_scripts
- 3.19.12. admin_head wp:action:admin_head
- 3.19.13. template_redirect wp:action:template_redirect
- 3.19.14. wp_head, wp_footer, get_footer wp:action:wp_head
- 3.19.15. add_meta_boxes, add_meta_boxespost_type wp:action:add_meta_boxes
- 3.19.16. edit_page_form wp:action:edit_page_form
- 3.19.17. edit_form_advanced
- 3.19.18. save_post wp:action:save_post
- 3.19.19. do_feed_$feedname wp:action:do_feed_$feedname
- 3.20. FTP setting wp:ftp
- 3.21. Sanitizing and Escaping User Data
- 3.22. Ajax wp:ajax
- 3.23. Theme wp:theme
- 3.24. Genesis Framework wp:genesis
- 3.24.1. Functions wp:genesis:functions
- 3.24.1.1. upgrade.php
- 3.24.1.2. compat.php
- 3.24.1.3. general.php
- 3.24.1.4. options.php wp:genesis:options
- 3.24.1.5. image.php
- 3.24.1.6. markup.php wp:genesis:functions:markup.php
- 3.24.1.7. breadcrumb.php
- 3.24.1.8. menu.php
- 3.24.1.9. layout.php wp:genesis:functions:layout.php
- 3.24.1.10. formatting.php
- 3.24.1.11. seo.php
- 3.24.1.12. widgetize.php
- 3.24.1.13. feed.php
- 3.24.1.14. toolbar.php
- 3.24.1.15. head.php
- 3.24.2. Structure
- 3.24.3. Life cycle
- 3.24.4. Actions and filters
- 3.24.5. Theme support wp:genesis:theme support
- 3.24.6. Post Type Supports wp:genesis:post type supports
- 3.24.7. Shortcodes wp:genesis:shortcodes
- 3.24.8. Child theme
- 3.24.9. Genesis Visual Hook Guide
- 3.24.1. Functions wp:genesis:functions
- 3.25. Customize API wp:api:customize
- 3.26. Admin
- 3.26.1. Toolbar - Admin bar wp:admin:toolbar
- 3.26.2. List Table API
WP_Posts_Lists_Tablewp:api:list table- 3.26.2.1. Action
manage_${post_type}_posts_custom_columnwp:api:list table:action:manage_postspost_type_custom_column - 3.26.2.2. Filter
manage_${taxonomy}_custom_columnandmanage_users_custom_columnwp:api:list table:filter:managetaxonomy_custom_column - 3.26.2.3. Filter
manage_${post_type}_posts_columnswp:api:list table:filter:managepost_type_posts_columns - 3.26.2.4. Filter
manage_{screen_id}_columnswp:api:list table:filter:managescreen_id_columns - 3.26.2.5. Filter
manage_{screen_id}_sortable_columnswp:api:list table:filter:managescreen_id_sortable_columns
- 3.26.2.1. Action
- 3.27. Custom Cache
- 3.28. Custom page
- 3.29. Database wp:db
- 3.30. Plugins
- 3.30.1. Custom Plugin
- 3.30.2. GitHub Updater wp:plugin:github-updater
- 3.30.3. Classic Editor wp:plugin:classic-editor
- 3.30.4. Development
- 3.30.5. Translation
- 3.30.6. Image
- 3.30.7. Image, Media in cloud
- 3.30.8. User, Roles, Capability
- 3.30.9. Fields
- 3.30.10. Post, Post Type
- 3.30.11. Backup, migrate, import, export
- 3.30.11.1. BackUpWordPress wp:plugin:backupwordpress
- 3.30.11.2. backupbuddy (iThemes) wp:plugin:backupbuddy
- 3.30.11.3. WP Migrate DB wp:plugin:wp-migrate-db
- 3.30.11.4. wp-migrate-db-pro, wp-migrate-db-pro-media-files
- 3.30.11.5. wp-all-import-pro, wp-all-export-pro
- 3.30.11.6. all-in-one-wp-migration wp:plugin:all-in-one-wp-migration
- 3.30.11.7. wordpress-importer
- 3.30.11.8. WP Sync DB wp:plugin:wp-sync-db
- 3.30.12. DB Optimize
- 3.30.13. WP Admin
- 3.30.14. Cache, CDN
- 3.30.15. Google
- 3.30.16. SEO
- 3.30.17. Social media, Push Notification
- 3.30.18. Post type - Comments
- 3.30.19. Post type - Events
- 3.30.20. Post type - Testimonial
- 3.30.21. Forms, Polls, Surveys
- 3.30.22. Site search
- 3.30.23. Redirect, Permalinks
- 3.30.24. Security
- 3.30.25. Authentication
- 3.30.26. Email
- 3.30.27. Menu, Nav, Breadcrumb
- 3.30.28. Slider, Lightbox
- 3.30.29. Widgets
- 3.30.30. Ecommerce
- 3.30.31. Theme
- 3.30.32. Page Builder
- 3.30.33. jetpack
- 3.31. Debug, WP_Error
- 3.32. Installation Language
- 3.33. File Permissions
- 3.34. SSL
- 3.35. REST API wp:rest
- 3.35.1. Methods (Also called operations, verbs, routes)
- 3.35.2. Status Codes
- 3.35.3. Global parameters
- 3.35.4. Pagination and Ordering
- 3.35.5. Response Header fields
- 3.35.6. Endpoints
- 3.35.7. Authentication
- 3.35.8. Current request is REST request?
- 3.35.9. New internal request
- 3.35.10. New external request and read response
- 3.35.11. Cache
- 3.35.12. Hooks
- 3.35.13. Basic Auth
- 3.35.14. Nonce
- 3.35.15. JWT - JSON Web Token
- 3.36. SOAP vs RESTful API
- 3.37. Life Cycle
- 3.38. PHP 7
- 3.39. Tools
- 3.40. UC: Simple HTML page with form submit
- 3.41. TS: Download failed. Destination directory for file streaming does not exist or is not writable
- 3.42. TS: Hacked wp:ts:hacked
- 3.43. TS: wp_deregister_script was called incorrectly
- 3.44. TS: Image editor not working, Edit Media, PHP open tag
- 3.45. TS: Check list
- 4. ColdFusion
- 5. Python
- 6. CSS
- 6.1. Display, Float, Position,
box-sizing - 6.2. Horizontal Center Absolute
- 6.3. Transform, Transition
- 6.4. Insert CSS File css:js insert
- 6.5. Font, Text css:font
- 6.5.1. Ellipsis
- 6.5.2. Long Word
- 6.5.3. Font size, line-height, letter-spacing
- 6.5.4. Prevent sup and sub affecting line height css:sup:line-height
- 6.5.5. Fluid Typography
- 6.5.6.
@font-facecss:font-face - 6.5.7.
font-feature-settingcss:font-feature-settings - 6.5.8.
font-variation-settingscss:font-variation-settings
- 6.6. CSS Selectors CSS Selectors
- 6.7. Specificity
- 6.8. Functions
- 6.9. border-image
- 6.10. background css:background
- 6.11. List style
- 6.12. Color
- 6.13. Gradient and repeat gradient
- 6.14. opacity
- 6.15. Shadow
- 6.16. object-fit
- 6.17. CSS target IE 11 and IE10
- 6.18. @media css:@media
- 6.19. @keyframes
- 6.20. @import
- 6.21. @viewport
- 6.22. @supports css:@supports
- 6.23. Flexbox
- 6.24. Equal width for unknown number of child elements
- 6.25. Grid
- 6.25.1. container: grid, grid-template
- 6.25.2. container: display, grid-template-columns, grid-template-rows
- 6.25.3. container: grid-template-areas css:grid:container:grid-template-areas
- 6.25.4. container: grid-column-gap, grid-row-gap, grid-gap css:grid:container:grid-gap
- 6.25.5. container: justify-items, item: justify-self css:grid:container:justify-items
- 6.25.6. container: align-items, item: align-self css:grid:container:align-items
- 6.25.7. container: justify-content css:grid:container:justify-content
- 6.25.8. container: align-content css:grid:container:align-content
- 6.25.9. container: grid-auto-columns, grid-auto-rows
- 6.25.10. container: grid-auto-flow css:grid-auto-flow
- 6.25.11. item: grid-column, grid-row css:grid:item:grid-column, grid-row
- 6.25.12. item: grid-area css:grid:item:grid-area
- 6.25.13. item: justify-self, align-self css:grid:item:justify-self css:grid:item:align-self
- 6.25.14. IE10, IE11
- 6.26. Horizontal and Vertical Center
- 6.27. Column for text
- 6.28. Shape
- 6.29. SVG
- 6.30. Sass css:scss css:sass
- 6.31. Use Cases
- 6.31.1. iOS Hover
- 6.31.2. Image inside a div
- 6.31.3. Transparent Image
- 6.31.4. Image scaling
- 6.31.5. Button Hover Double Quotes
- 6.31.6. Text Label
- 6.31.7. Button with icon on the left
- 6.31.8. Vertical Line Separator in <li>
- 6.31.9. Wrap h1 around a div on the right, float right sequence
- 6.31.10. 100% width and with margin
- 6.31.11. Align text with elements
- 6.31.12. Circle Number List
- 6.31.13. superscript, subscript uneven line height
- 6.31.14. Swap image in
@media - 6.31.15. Hightlight element when the hash and id of element match
- 6.31.16. Go to anchor element with position:fixed header
- 6.31.17. Center
position:absolute
- 6.32. BEM
- 6.33. Houdini
- 6.34. CSS Frameworks
- 6.34.1. Bootstrap
- 6.34.2. Tailwind
- 6.34.2.1. Install
- 6.34.2.2. Config
- 6.34.2.3. padding
- 6.34.2.4. About turning off preflight
- 6.34.2.5. Pseudo-class Reference
- 6.34.2.6. Position modifiers
- 6.34.2.7. Form modifiers
- 6.34.2.8. Child Modifiers and Sybling Modifiers
- 6.34.2.9. Attribute modifier
- 6.34.2.10. Media modifier
- 6.34.2.11. Dark mode
- 6.34.2.12. Arbitrary Values
- 6.34.2.13. Tools built on top of Tailwind
- 6.34.3. MUI
- 6.34.4. Pico.css
- 6.34.5. Animate.css
- 6.1. Display, Float, Position,
- 7. HTML
- 7.1. DOM
- 7.1.1. Window
- 7.1.2. Document Object
- 7.1.3. NodeList Object NodeList
- 7.1.4. Element Object
- 7.1.5. Event Object
- 7.1.5.1. Basics
- 7.1.5.2. Dispatch event, trigger event
- 7.1.5.3. onclick
- 7.1.5.4. onchange
- 7.1.5.5. Events about checkbox
- 7.1.5.6. Events about radio
- 7.1.5.7. MouseEvent Object
- 7.1.5.8. KeyboardEvent Object
- 7.1.5.9. HashChangeEvent Object
- 7.1.5.10. PageTransitionEvent
- 7.1.5.11. FocusEvent Object
- 7.1.5.12. AnimationEvent Object
- 7.1.5.13. HTML DOM Event
- 7.1.5.14. readystatechange html:event:readystatechange
- 7.1.5.15. MessageEvent
- 7.1.6. DOMTokenList Object
- 7.1.7. NamedNodeMap Object
- 7.1.8. Attr Object
- 7.1.9. PointerEvent
- 7.2. Global Attributes
- 7.3. Events and Event Attributes
- 7.4. Canvas
- 7.5. Tags
- 7.5.1. Tags
- 7.5.2.
<a> - 7.5.3.
<iframe>attributesandbox - 7.5.4.
<picture>,<img srcset sizes> - 7.5.5.
<dl>,<dt>,<dd>Definition Term/name in a Definition list - 7.5.6.
<blockquote>and<q> - 7.5.7.
<time> - 7.5.8.
<figure><figcaption> - 7.5.9.
<video>html:video - 7.5.10.
<audio> - 7.5.11.
<tbody>,<thead>and<tfoot> - 7.5.12.
<input> - 7.5.13.
<table>
- 7.6. Meta
- 7.7. Header
- 7.7.1. Age header:age
- 7.7.2. Cache-Control, Pragma - response header:cache-control
- 7.7.3. Expires header:expires
- 7.7.4. Content-Security-Policy (CSP) - response
- 7.7.5. X-Frame-Options
- 7.7.6. Content-Type
- 7.7.7. X-Content-Type-Options - response
- 7.7.8. Content-Disposition
- 7.7.9. X-XSS-Protection - response
- 7.7.10. Referer - request header:referer
- 7.7.11. Referrer-Policy - response header:referrer-policy
- 7.7.12. Strict-Transport-Security - response
- 7.7.13. Vary - response header:vary
- 7.7.14. Link header:link
- 7.7.15. Accept-CH - response header:accept-ch
- 7.7.16. X-Powered-By
- 7.8. MIME
- 7.9. Cookie
- 7.10. Email
- 7.10.1. View email source on Outlook
- 7.10.2. Doctype
- 7.10.3. Guidelines
- 7.10.4. Responsive
- 7.10.5. Even 3 columns
- 7.10.6. My even 2 columns
- 7.10.7. My template
- 7.10.8. Hybrid Responsive Design
- 7.10.9. Inbox Preview
- 7.10.10. Gmail Image Caching
- 7.10.11. CSS and HTML Support across Devices email:css:support
- 7.10.12. MJML
- 7.10.12.1. Install & Basics
- 7.10.12.2. mjml mjml:tag:mjml
- 7.10.12.3. mj-head mjml:tag:mj-head
- 7.10.12.4. mj-head > mj-preview mjml:tag:mj-preview
- 7.10.12.5. mj-head > mj-attributes mjml:tag:mj-attributes
- 7.10.12.6. mj-head > mj-breakpoint
- 7.10.12.7. mj-head > mj-style mjml:tag:mj-style
- 7.10.12.8. mj-head > mj-font mjml:tag:mj-font
- 7.10.12.9. mj-head > mj-title mjml:tag:mj-title
- 7.10.12.10. mj-head > mj-body > mj-section (mj-container)
- 7.10.12.11. mj-body > mj-include mjml:tag:mj-include
- 7.10.12.12. mj-body > mj-section mjml:tag:mj-section
- 7.10.12.13. mj-wrapper > mj-section mjml:tag:mj-wrapper
- 7.10.12.14. mj-container > mj-hero mjml:tag:mj-hero
- 7.10.12.15. mj-section > mj-column mjml:tag:mj-column
- 7.10.12.16. mj-elements inside mj-column
- 7.10.12.17. mj-group to group columns
- 7.10.12.18. Background Image
- 7.10.12.19. Change default padding
- 7.10.12.20. Caveats
- 7.10.13. Non breaking css:content
- 7.10.14. Remove spaces between HTML end tags and start tags email:spaces between html tags
- 7.10.15. Don't use anchor link!
- 7.11. Phone number
- 7.12. Link html:link
- 7.12.1.
media="print"html:link:media - 7.12.2.
rel="preload"html:link:rel:preload - 7.12.3.
rel="prefetch" - 7.12.4.
rel="dns-prefetch" - 7.12.5. preload vs prefetch and Network Prioritisation
- 7.12.6.
rel="canonical"andrel="alternate"html:link:rel:canonical html:link:rel:alternate - 7.12.7.
rel="manifest.json"html:link:rel:manifest - 7.12.8.
rel="icon"rel="apple-touch-startup-image"rel="apple-touch-icon" - 7.12.9.
rel="noopener"html:link:rel:noopener
- 7.12.1.
- 7.13. Autofill, Input Types
- 7.14. Special Characters css:content
- 7.15. Geolocation
- 7.16. Storage
- 7.17. Application Cache
- 7.18. ARIA html:ARIA
- 7.19. Display price
- 7.20. Progressive Web App
- 7.21. AMP google:amp
- 7.21.1. Search Engine Discovery
- 7.21.2. AMP JS Library
- 7.21.3. Boilerplate
- 7.21.4.
amp-customamp:amp-custom - 7.21.5.
amp-imageamp:amp-image - 7.21.6.
amp-bindamp-bind-macroamp:amp-bind - 7.21.7.
amp-carouselamp:amp-carousel - 7.21.8.
amp-videoamp:amp-video - 7.21.9.
amp-position-observeramp:amp-position-observer - 7.21.10. amp-animation amp:amp-animation
- 7.21.11. Actions and Events amp:event
- 7.21.12. Layout & media queries amp:layout
- 7.21.13. amp-analytics amp:amp-analytics
- 7.21.14. AMPHTML ads
- 7.1. DOM
- 8. JavaScript
- 8.1. Loading Sequence js:defer js:async
- 8.2. External Script,
document.writejs:external script - 8.3. Initialization >
- 8.4. Variable Scope
- 8.5. Closure, Javascript Singleton
- 8.6. Types, typeof, constructor
- 8.6.1. Number
- 8.6.2. Boolean
- 8.6.3.
undefinedundefined or has not been assigned a value - 8.6.4.
null- a special type of object - 8.6.5. String
- 8.6.6. Array
- 8.6.6.1. Array.xxx() vs Array.prototype.xxx()
- 8.6.6.2. DOM elements
HTMLCollection - 8.6.6.3. Numeric array key exists
- 8.6.6.4. Variable is a non-empty array
- 8.6.6.5. array.indexOf(), array.prototype.includes()
- 8.6.6.6. Array.prototype.pop(), Array.prototype.shift(), array.push(), array.unshift()
- 8.6.6.7.
array.concat()merge without altering nor removing dups, return a new array - 8.6.6.8. array.slice() vs array.splice()
- 8.6.6.9. array.join(), array.toString(), array.valueOf()
- 8.6.6.10. array.sort(), array.reverse()
- 8.6.6.11. array loop: for, array.forEach()
- 8.6.6.12. array.map(), array.filter()
- 8.6.6.13. array.every(), array.some()
- 8.6.6.14. Array.prototype.reduce()
- 8.6.7. Date
- 8.6.8. Math js:Math
- 8.6.9. RegExp
- 8.6.10. Function
- 8.6.11. Exception, Error
- 8.6.12. Prototype Object js:prototype
- 8.6.13. Symbol js:symbol
- 8.6.14. Type Conversion
- 8.7. Ternary
- 8.8. Loops and Breaks
- 8.9. Enumerable js:Enumerable
- 8.10. JSONP, JSON
- 8.11. DOM Manipulation
- 8.12. URI and URI Component Encode, Current URL, open, email link
- 8.13. Cookie js:cookie
- 8.14. Object only, no Class js:Object
- 8.14.1. Define an object
- 8.14.2. Contructor
new,Object.create - 8.14.3. Methods of
Objectconstructor - 8.14.4. instanceof js:instanceof
- 8.14.5.
Object.definePropertyjs:Object.defineProperty - 8.14.6.
object.assginObject assignment - 8.14.7. Compare objects without methods (JSON-style)
- 8.14.8. Reuse previous value when declaring
- 8.14.9. Example
- 8.15. Extending Native Object
- 8.16. Classical Inheritance, Parasitic Inheritance js:inheritance
- 8.17. OOP Pattern
- 8.18. Promise
- 8.19. Strict mode
- 8.20.
thisandself - 8.21. Observable js:observable
- 8.22. Worker
- 8.23. XMLHttpRequest
- 8.24. File handling
- 8.25. Apply XSLT on XML
- 8.26. Apply XPath
- 8.27. Web APIs
- 8.28. Testing
- 8.29. Worker
- 8.30. iFrame
- 8.31. setTimeout recall
- 8.32. Device Detection
- 8.33. CSS Media Viewport Size js:viewport size
- 8.34. ECMAScript
- 8.35. TypeScript
- 8.36. Use Cases
- 9. JavaScript projects
- 9.1. Vue.js
- 9.1.1. Basics
- 9.1.2. Data and methods
vm.datavue:data - 9.1.3. Instance Lifecycle Hooks
- 9.1.4. Component
- 9.1.4.1. Basics
- 9.1.4.2. DOM Template Parsing Caveats
- 9.1.4.3. Option data has to be a function
- 9.1.4.4. Option filters
- 9.1.4.5. Props down
- 9.1.4.6. Events Up
- 9.1.4.7. DOM template
- 9.1.4.8. Except class and style :: parent attributes overwrite child component
- 9.1.4.9. Slots
- 9.1.4.10. Dynamic component, v-bind:is, <component>, <keep-alive>
- 9.1.4.11. Get Child Component One Time
- 9.1.4.12. Async Components
- 9.1.4.13. Recursive and Circular Reference
- 9.1.4.14. inline-template vue:inline-template
- 9.1.4.15. X-Templates vue:x-template
- 9.1.5. Template Syntax
- 9.1.5.1. String template vs DOM template
- 9.1.5.2.
v-once - 9.1.5.3.
v-htmlRaw HTML - 9.1.5.4.
v-bind:attributeName,:attributeName - 9.1.5.5. Expression
- 9.1.5.6. Directive Arguments
v-bind:href,v-bind:class,v-on:click - 9.1.5.7. Dynamic Directive Argument
- 9.1.5.8. Modifiers
v-on:submit.prevent - 9.1.5.9.
v-if,keyattribute,v-show - 9.1.5.10.
v-for - 9.1.5.11. Event, v-on, @attributeName
- 9.1.5.12. Form input, v-model
- 9.1.6. Render Functions & JSX
- 9.1.7.
vm.computedproperty,vm.watchproperty - 9.1.8. Transition & Animation
- 9.1.9. Mixins
- 9.1.10. Custom Directive
- 9.1.11. Plugin vue:plugin
- 9.1.12. Production Development
- 9.1.13. vue-router vue:plugin:router vue:router
- 9.1.13.1. <router-view>, <router-link>, option:routes, this.$router, this.$route
- 9.1.13.2. option:routes, $route.params, regex, option:routes[]:name, option:routes[]:component
- 9.1.13.3. option:routes[]:components, multiple <router-view>'s
- 9.1.13.4. option:routes[]:redirect, option:routes[]:alias
- 9.1.13.5. Route object, $route, this.$route, option:scrollBehavior vue:router:route
- 9.1.13.6. option:scrollBehavior
- 9.1.13.7. option:routes[]:props, component:props
- 9.1.13.8. React to params changes, component:watch, component:beforeRouteUpdate
- 9.1.13.9. Nested routes, <router-view>, option:routes:children
- 9.1.13.10. Programmatic Navigation, this.$router,
router.* - 9.1.13.11. option:mode
- 9.1.13.12. Guards
- 9.1.13.13. Fetch data
- 9.1.13.14. Lazy loading
- 9.1.14. State Management, Vuex
- 9.1.14.1. Basics, option:state
- 9.1.14.2. mapState in component computed props
- 9.1.14.3. option:getters, mapGetters in component computed props
- 9.1.14.4. option:mutations, commit/mapMutations in component, Synchronous vue:plugin:vuex:mutations
- 9.1.14.5. option:actions with context, dispatch/mapActions in component, Asynchronous
- 9.1.14.6. option:modules
- 9.1.14.7. option:strict, strict mode
- 9.1.14.8. option:watch
- 9.1.14.9. option:plugin
- 9.1.14.10. App Structure, Example
- 9.1.15.
vue-clivue:vue-cli - 9.1.16. Dev - Webpack Template
- 9.1.17. Ajax, axios
- 9.2. jQuery
- 9.2.1. Version, Closure, document ready
- 9.2.2. Ajax
- 9.2.3. jQuery Selectors, Loop through found elements
- 9.2.4. iFrame refer to parent
- 9.2.5. Find elements
- 9.2.6. Pass
$(this)to javascript function - 9.2.7. General click to element
- 9.2.8. Pointer Events
- 9.2.9. Replace class
- 9.2.10. Move element
- 9.2.11. Add Style
- 9.2.12. Form
- 9.2.13. Animate
- 9.2.14. Viewport Change Event
- 9.2.15. jQuery UI: Dialog jqueryui:dialog
- 9.2.16. jQuery UI: Sortable
- 9.3. TweenLite, TweenMax, Greensock
- 9.4. Select2 js:lib:select2
- 9.4.1. Basic
- 9.4.2. Internal Option data, <optgroup>
- 9.4.3. config:ajax
- 9.4.4. config:data
- 9.4.5. config:templateResult, Template
- 9.4.6. config:templateSelection, Change display of selected value
- 9.4.7. config:maximumSelectionLength
- 9.4.8. config:placeholder
- 9.4.9. config:allowClear
- 9.4.10. config:width
- 9.4.11. config:tags
- 9.4.12. config:tokenSeparators
- 9.4.13. config:createTag
- 9.4.14. config:insertTag
- 9.4.15. config:matcher, config:minimumInputLength, config:minimumResultsForSearch, Search
- 9.4.16. Methods, add, select, clear
- 9.4.17. Method, get
- 9.4.18. Method, open|close dropdown, if initialized, destroy
- 9.4.19. Example: Destroy and Initiate
- 9.4.20. Add down arrow to multiple <select>
- 9.4.21. Events
- 9.5. Owl Carousel
- 9.6. Fine Uploader
- 9.7. Dropzone
- 9.7.1. Basics
- 9.7.2. option:function:accept(file, done)
- 9.7.3. option:previewsContainer null|string
- 9.7.4. option:autoProcessQueue
- 9.7.5. option:previewTemplate String, option:dict*
- 9.7.6. option:addRemoveLInks null:true
- 9.7.7. option:translation
- 9.7.8. option:maxFiles int
- 9.7.9. option:maxFilesize int (MB)
- 9.7.10. option:event:init, add events
- 9.7.11. option:clickable null|true|false|an html element|a CSS selector|array of html
- 9.7.12. option:acceptedFiles string of list
- 9.7.13. option:maxThumbnailFilesize (MB), thumbnailWidth, thumbnailHeight, thumbnailMethod
- 9.7.14. event:addedfile
- 9.7.15. event:removedfile(file)
- 9.7.16. event:thumbnail(file, dataUrl)
- 9.7.17. event:processing(file)
- 9.7.18. event:uploadprogress(file, progress, bytesSent)
- 9.7.19. event:error(file)
- 9.7.20. event:success(file, responseText)
- 9.7.21. event:complete(file)
- 9.7.22. event:canceled(file)
- 9.7.23. event:maxfilesexceeded(file)
- 9.7.24. event:maxfilesreached(file)
- 9.7.25. FAQ
- 9.7.26. Show error returned by server
- 9.7.27. Show server response
- 9.7.28. Show files already stored on server
- 9.7.29. Sortable
- 9.8. modernizr
- 9.8.1. Feature
- 9.8.2. Option
- 9.8.3. API
- 9.8.3.1. Modernizr.on(feature, cb)
- 9.8.3.2. .addTest('customFeatureName', cb)
- 9.8.3.3. .atRule(prop)
- 9.8.3.4. ._domPrefixes, ._prefixes
- 9.8.3.5. .hasEvent(eventName, [element])
- 9.8.3.6. .prefixedCSS(prop), .prefixed(prop, [obj], [elem])
- 9.8.3.7. .prefixedCSSValue(prop,value)
- 9.8.3.8. .testAllProps(prop,[value], [skipValueTest]), .testProp(prop,[value], [useValue])
- 9.9. UAParser.js
- 9.10. flowpaper
- 9.11. reveal.js
- 9.12. VelocityJS
- 9.13. BabylonJS
- 9.14. Angular
- 9.14.1. Who use Angular?
- 9.14.2. 1.x AngularJS
- 9.14.3. 2.x Angular
- 9.14.3.1. QuickStart seed
- 9.14.3.2. SystemJS vs Webpack
- 9.14.3.3. package.json
- 9.14.3.4. tsconfig.json, d.ts, lib.d.ts
- 9.14.3.5. lite-server npm:lite-server
- 9.14.3.6. CLI
- 9.14.3.7. Polyfills
- 9.14.3.8. Bootstrap ng:bootstrap
- 9.14.3.9. Root Module ng:root module
- 9.14.3.10. ngModule
- 9.14.3.11. Component ng:component
- 9.14.3.12. Template
- 9.14.3.13. Pipe ng:pipe
- 9.14.3.14. Directive ng:directive
- 9.14.3.15. Model and Custom Component
- 9.14.3.16. Form
- 9.14.3.17. Service, Dependency Injection
- 9.14.3.18. Route
- 9.14.3.19. API
- 9.14.3.20. Style Guide
- 9.15. React.js
- 9.15.1. CDN
- 9.15.2. create-react-app
- 9.15.3. Vite
- 9.15.4. react-bootstrap
- 9.15.5. react-router-dom
- 9.15.6. JSX
- 9.15.7. TypeScript
- 9.15.8. CSS
- 9.15.9. Redux
- 9.15.10. Custom hooks
- 9.15.11. Use absolute imports
- 9.15.12. Refactor
- 9.15.13. Incrementally Migrate to React
- 9.15.14. React State data structure
- 9.15.15. Directory Structure
- 9.15.16. Tools
- 9.16. CryptoJS NPM
- 9.17. CryptoJS Goolge Code Archive
- 9.18. Moment.js
- 9.1. Vue.js
- 10. NodeJS
- 10.1. Install
- 10.2. Basic
- 10.3. Global Object
- 10.4. Modules, Packages, Frameworks
- 10.4.1. Get version
- 10.4.2. Semver
~vs^ - 10.4.3. package.json
- 10.4.4. Update Package
- 10.4.5. Scoped package
- 10.4.6. Peer dependencies
- 10.4.7. Core modules
- 10.4.8. npx
- 10.4.9. BabelJS
- 10.4.10. axios
- 10.4.11. lodash nodejs:lodash
- 10.4.12. lighthouse npm:lighthouse
- 10.4.13. font-awesome npm:font-awesome
- 10.4.14. prismjs npm:prismjs
- 10.4.15. bootstrap npm:bootstrap
- 10.4.16. browser-sync npm:browser-sync
- 10.4.17. concurrently npm:concurrently
- 10.4.18. concurrent-transform npm:concurrent-transform
- 10.4.19. gulp npm:gulp
- 10.4.19.1. Example
- 10.4.19.2. Pass parameters from CLI to Gulp
- 10.4.19.3.
gulp.src,gulp.dest - 10.4.19.4. gulp-debug
- 10.4.19.5. gulp-plumber
- 10.4.19.6. gulp-remember
- 10.4.19.7. gulp-cache
- 10.4.19.8. gulp-sort
- 10.4.19.9. gulp-concat
- 10.4.19.10. gulp-rename
- 10.4.19.11. gulp-sourcemaps
- 10.4.19.12. gulp-filter
- 10.4.19.13. gulp-sass npm:gulp-sass
- 10.4.19.14. gulp-merge-media-queries
- 10.4.19.15. gulp-line-ending-corrector
- 10.4.19.16. gulp-uglifycss
- 10.4.19.17. gulp-postcss npm:gulp-postcss Doc
- 10.4.19.18. gulp-uglify
- 10.4.19.19. gulp-babel
- 10.4.19.20. gulp-responsive
- 10.4.19.21. gulp-image-resize npm:gulp-image-resize
- 10.4.19.22. gulp-imagemin
- 10.4.19.23. gulp-svg-sprite
- 10.4.19.24. gulp-notify
- 10.4.20. grunt
- 10.4.21. webpack
- 10.4.22. PostCSS, postcss-cli cssnano autoprefixer
- 10.4.23. co-fs, co nodejs:co-fs nodejs:co
- 10.4.24. http, https nodejs:http
- 10.4.25. cors
- 10.4.26. xml2js
- 10.4.27. casual nodejs:casual
- 10.4.28. crypto - Built-in nodejs:crypto
- 10.4.29. body-parser
- 10.4.30. httpster
- 10.4.31. express nodejs:express
- 10.4.32. sails
- 10.4.33. koa
- 10.4.34. Promise
- 10.4.35. bluebird nodejs:bluebird
- 10.4.36. lite-server npm:lite-server
- 10.4.37. node-dev, nodemon, reload
- 10.4.38. jshint
- 10.4.39. mongoose nodejs:mongoose
- 10.4.40. sequelize nodejs:sequelize
- 10.4.41. ws
- 10.4.42. socket.io
- 10.4.43. mocha, chai, nock, rewire, sinon
- 10.4.44. supertest, cheerio
- 10.4.45. istanbul
- 10.4.46. svgo nodejs:svgo
- 10.4.47. serve
- 10.5. Custom Module
- 10.6. node-gyp node-gyp
- 10.7. NPM CLI
- 10.8. Increase memory and
NODE_OPTIONSnodejs:memory - 10.9. yarn
- 10.10. TS: gulp-util is deprecated
- 10.11. TS: pathspec
- 10.12. TS: Unmet dependencies
- 10.13. TS: ENOSPC: System limit for number of file watchers reached
- 11. Prerender.io
- 12. Social Media
- 13. SEO & Marketing
- 13.1. Starter Guide
- 13.2. Google Test Mobile Speed
- 13.3. Google My Business
- 13.4. Google Analytics google:ga
- 13.4.1. Limits, Versions, Accounts
- 13.4.2. Views, Filters, Segments
- 13.4.3. Dimensions vs Metrics ga:dimensions ga:metrics ga:scope
- 13.4.4. Bounce rate
- 13.4.5. Audience
- 13.4.6. Acquisition ga:channel
- 13.4.7. Behavior
- 13.4.8. Conversions
- 13.4.9. Add Google Search Console to a property
- 13.4.10. Campaigns and Goals track across user session
- 13.4.11. Goal vs Event
- 13.4.12. Segment ga:segment
- 13.4.13. Tracking Code - gtag.js, analytics.js ga:gtag
- 13.4.13.1. Basics
- 13.4.13.2. Custom parameters for dynamic remarketing ga:dynamic remarketing
- 13.4.13.3. gtag.js
configanalytics.jscreatega:gtag:config ga:ga:create - 13.4.13.4. gtag.js
setga:gtag:set - 13.4.13.5. gtag.js
eventga:gtag:event - 13.4.13.6. Send to multiple property ids ga:gtag:event:send_to
- 13.4.13.7. ga Command Queue and Object Methods
- 13.4.13.8. ga Command Queue
- 13.4.13.9. ga Object Method
- 13.4.14. Tracker Object Methods
- 13.4.15. fieldsObject ga:fieldsObject
- 13.4.16. Regular Expressions ga:regex
- 13.4.17. Analytics Solutions Gallery
- 13.4.18. Email tracking, Measurement Protocol
- 13.4.19. Site Search
- 13.4.20. Custom Alerts
- 13.4.21. Google Analytis Core Reporting API
- 13.4.22. Remarketing with Analytics ga:remarketing
- 13.4.23. Dynamic Remarketing with Analytics ga:dynamic remarketing
- 13.4.24. AMP ga:amp
- 13.4.25. Transfer a property
- 13.4.26. Track 404 pages
- 13.4.27. TS: Too Many Self-referrals
- 13.4.28. Learn
- 13.4.29. Checklist
- 13.5. Google Optimize google:optimize
- 13.6. GA Dev Tools: Query Explorer
- 13.7. GA Dev Tools: Campaign URL Builder google:campaign-url-builder
- 13.8. Google Search Console google:search console
- 13.9. Google Trends
- 13.10. Google Shopping Insights
- 13.11. Google Ads - Google AdWords, Google Keyword Planner, Google Video Advertising
- 13.11.1. Keyword Planner google:ads:keyword-planner
- 13.11.2. Google Ads - AdWords
- 13.11.2.1. Campaign Goals
- 13.11.2.2. Keywords Match Type adwords:keywords match types
- 13.11.2.3. Negative keywords
- 13.11.2.4. Campaigns, Ad groups
- 13.11.2.5. Placement
- 13.11.2.6. Quality Score, Ad Rank, Ad Position
- 13.11.2.7. Bidding Strategy - Media Cost Model
- 13.11.2.8. Networks, Campaign Types, Ad Formats
- 13.11.2.9. Targeting options adwords:targeting
- 13.11.2.10. Ad Group Automated Targeting
- 13.11.2.11. Ad Formats in networks
- 13.11.2.12. Ad extension
- 13.11.2.13. Ad customizers
- 13.11.2.14. ValueTrack URL parameters
- 13.11.2.15. Supported Ad Sizes
- 13.11.2.16. Conversion tracking
- 13.11.2.17. Remarketing Audience, Customer Match
- 13.11.2.18. Reports
- 13.11.2.19. Autotagging google:ads:auto-tagging
- 13.11.2.20. Automated Rules
- 13.11.2.21. Third party impression tracking
- 13.11.2.22. Accounts google:ads:account
- 13.11.2.23. Link a Google Analytics Property and Google Ads accounts google:ads:link GA
- 13.11.3. Make ads
- 13.11.4. Google Video Advertising
- 13.11.5. Google Rich Media Gallery
- 13.12. Google Tag Manager google:gtm
- 13.13. Google Surveys google:surveys
- 13.14. Google Marketing Platform google:marketing platform
- 13.15. Google SERP Features
- 13.16. Website Keywords
- 13.17. Site Audit Tools - check js errors, GA tagging, etc.
- 13.18. SEO, Search PPC Strategy Tools and Services
- 13.19. Conversion Rate Optimization - CRO
- 13.20. Remarketing vs Retargeting
- 13.21. Competitive Marketing
- 13.22. Optimize Images
- 13.23. Rich snippet, JSON LD google:json-ld
- 13.24. robots.txt
- 13.25. Disavow.txt
- 13.26. Design & User Experience
- 13.27. hreflang html:hreflang
- 13.28. Google News
- 13.29. Privacy Policy
- 14. UX and Design
- 15. Linux
- 15.1. Explain Shell
- 15.2. System Version, Distro
- 15.3. Environment and Shell Variables
- 15.4. Bash - Bourne-Agin Shell
- 15.5. zsh
- 15.6. User Management, Superuser, sudo
- 15.7. SSH, scp, sshpass
- 15.7.1. Install SSH Server
- 15.7.2. Create
.sshfolder - 15.7.3. Generate client public and private keys, transfer public key to remote
- 15.7.4. github:ssh:key:fingerprint
- 15.7.5. Config linux:ssh:config
- 15.7.6. Remove a known_host
- 15.7.7. Use private key from SSH server to connect
- 15.7.8. Restart SSH
- 15.7.9. sshd_config SSH Daemon Config File
- 15.7.10. Disallow SSH as root user, SSH Daemon Config linux:ssh:disallow_root
- 15.7.11. Disable Password Authentication
- 15.7.12. Login without setting public keys bash:sshpass
- 15.7.13. Random password
- 15.7.14. Run command
- 15.7.15. SSH tunneling
- 15.7.16. TS - WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
- 15.8. fail2ban linux:fail2ban
- 15.9. System
- 15.10. Filesystem
- 15.10.1.
/dev/mnt/media - 15.10.2. bindfs linux:bindfs
- 15.10.3. /etc system configuration files
- 15.10.4. Disk space
- 15.10.5. mkdir
- 15.10.6. File Permission
- 15.10.7. Links
- 15.10.8. Search files or directories, sort by size
- 15.10.9. Search Text in Files
- 15.10.10. Translate
- 15.10.11. column bash:column
- 15.10.12. cut
- 15.10.13. awk bash:awk
- 15.10.14. Folder structure
- 15.10.15. Compare files in 2 directories
- 15.10.16. Compare text files
- 15.10.17. rm
- 15.10.18. ftp bash:ftp
- 15.10.19. cp
- 15.10.20. mv
- 15.10.21. mount
- 15.10.22. tar
- 15.10.23. rsync
- 15.10.1.
- 15.11. Processes
- 15.12. Memory
- 15.13. Service
- 15.14. Network
- 15.14.1. /etc/network/interfaces file
- 15.14.2. DNS
- 15.14.3. Flush DNS
- 15.14.4. Get Server's Public IP linux:get public ip
- 15.14.5. CIDR Netmask (Network Mask) CIDR netmask
- 15.14.6. Port
- 15.14.7. Wireless network adapters
iwconfig - 15.14.8. Gather monthly bandwidth
- 15.14.9. Ping
- 15.14.10. Netcat
- 15.14.11. Test email address if valid without sending
- 15.15. Firewall linux:firewall linux:ufw
- 15.16. Timezone, NTP
- 15.17. Font linux:font
- 15.18. Hardware
- 15.19. Package, Repository
- 15.20. curl, wget
- 15.20.1. curl
- 15.20.1.1. Basic
- 15.20.1.2. Return HTTP response header only
- 15.20.1.3. Download with HTTPS or HTTP authentication
- 15.20.1.4. cURL download only after or before a date
- 15.20.1.5. HTTP POST
- 15.20.1.6. Use different IP address from DNS
- 15.20.1.7. Fetch web pages and save as HTML files
- 15.20.1.8. Fetch as Googlebot curl:ua:googlebot
- 15.20.1.9. Options
- 15.20.2. wget bash:wget
- 15.20.1. curl
- 15.21. Bash File, Input and Output
- 15.21.1. Version
echo $BASH_VERSION - 15.21.2. Shebang
- 15.21.3. Last directory
~- - 15.21.4. Directory of the curren script file
- 15.21.5. range bash:range
- 15.21.6. File Descriptors, Redirection Operator, Piping
- 15.21.7. declare bash:declare
- 15.21.8. Array bash:array
- 15.21.9.
echo,print - 15.21.10. printf bash:printf
- 15.21.11. Here document, here string bash:heredoc
- 15.21.12. Read command output line by line, Prompt
- 15.21.13. Date format
- 15.21.14. Last executed command
- 15.21.15. Passing arguments to command bash:pass arguments
- 15.21.16. String Operators
- 15.21.17. String comparison
- 15.21.18. Numeric comparison
- 15.21.19. Bash: shell variable with default value
- 15.21.20. if bash:if
- 15.21.21. Function bash:function
- 15.21.22. Special Parameters
- 15.21.1. Version
- 15.22. Cron
- 15.23. InCron - inotify cron
- 15.24. upx linux:upx
- 15.25. Commands
- 15.25.1. xclip bash:xclip
- 15.25.2. screen bash:screen
- 15.25.3. Run Sequentially
- 15.25.4. ls
- 15.25.5. less
- 15.25.6. head, tail
- 15.25.7. man
- 15.25.8.
cd -Navigate to previous directory (back) - 15.25.9. Word Count
wc - 15.25.10. sort
- 15.25.11. uniq
- 15.25.12. Replace String, Stream Editor
sed - 15.25.13. xargs
- 15.25.14. test - Conditionally run
- 15.25.15. Concatenate String with File
- 15.25.16. Change Encoding
- 15.25.17. Comment out every line in a file and save as a new file
- 15.25.18. sudo
- 15.25.19. SHA, salt hashing image:lucee:salt
- 15.25.20. byobu
- 15.25.21. Command line web browser
w3m - 15.25.22. set bash:set
- 15.25.23. Calendar
- 15.25.24. Calculator
- 15.25.25. expr
- 15.25.26. unit converter
- 15.25.27. shutdown
- 15.26. Cheat
- 15.27. Makefile
- 15.28. Nano
- 15.29. Vim
- 15.30. Ubuntu
- 15.31. Ubuntu Personal Package Archive - PPA
- 15.32. Production Server Setup
- 16. Domain Server, DNS Server, SSL/https
- 16.1. DNS records: A AAAA ANAME CNAME TXT ALIAS URL
- 16.2. MX record dns:mx
- 16.3. SPF record dns:spf
- 16.4. DKIM record dns:dkim
- 16.5. SOA record
- 16.6. redirect.center, redirect.name
- 16.7. Domain Forwarding
- 16.8. Domain Transfer
- 16.9. Addon domain
- 16.10. Park domain
- 16.11. Domain Registrars and Tools
- 16.12. DNS History, IP location, Website Hosting Finder
- 16.13. AXFR and Find subdomains
- 16.14. AutoSSL in WHM/cPanel autossl:whm
- 16.15. Install SSL
- 16.16. SSL Testing Tools
- 17. Docker
- 17.1. Version, Info, Installation
- 17.2. Command Help
docker docker-subcommand --help - 17.3. Images
- 17.4. run
- 17.4.1. Interactive mode with tty
-it - 17.4.2. Run an image in a container and keep running in background
- 17.4.3. Run an image in a temp container only once, then remove the container
- 17.4.4. Link docker:link
- 17.4.5. –name and –hostname
- 17.4.6. –privileged docker:run:privileged
- 17.4.7. Port
-p 1234:3000 - 17.4.8.
--platform linux/x86_64 - 17.4.9.
-v,--volumevs--mountvolume
- 17.4.1. Interactive mode with tty
- 17.5. Containers cp logs stats start stop restart rm inspect
- 17.6. Network
- 17.7. My Network
- 17.8. Volume
- 17.9. Hosts file
- 17.10. Dockerfile
- 17.11. Docker Compose
- 17.12. Docker for Windows
- 17.13. Docker Cloud
- 17.14. Docker registry
- 17.15. Common DevOps
- 17.16. Kubernetes - K8s
- 17.17. Images
- 17.17.1. image:alpine
- 17.17.2. image:debian
- 17.17.3. image:buildpack-deps:jessie-scm
- 17.17.4. image:buildpack-deps:stretch-curl
- 17.17.5. image:php:5.6.31-apache-jessie image:php:apache
- 17.17.6. image:php53
- 17.17.7. image:php:7
- 17.17.8. image:php:7-fpm
- 17.17.9. image:python
- 17.17.10. image:mongo
- 17.17.11. image:mysql
- 17.17.12. image:wordpress image:wordpress:apache
- 17.17.13. image:drupal image:drupal:apache
- 17.17.14. image:tatemz/wp-cli image:tatemz/wp-cli
- 17.17.15. image:wordpress:cli image:wordpress:cli
- 17.17.16. image:lucee docker:image:lucee
- 17.17.16.1. docker-compose.yml Dockerfile
- 17.17.16.2. Add ucanaccess db driver to TomCat, Add M$ Access Datasources
- 17.17.16.3. Lucee server and web settings
- 17.17.16.4. Custom tags (.cfm|.cfc) under
/opt/lucee/web/customtags/ - 17.17.16.5. Change Lucee Server Admin UI password
- 17.17.16.6. How does it work with Nginx?
- 17.17.16.7. Virtual Directory and TomCat
- 17.17.16.8. Add hosts
- 17.17.16.9. Scheduled Tasks
- 17.17.17. image:tomcat docker:image:tomcat
- 17.17.18. image:goaccess docker:image:goaccess
- 17.17.19. image:stilliard/pure-ftpd docker:image:ftp
- 17.17.20. image:dockerfile-from-image
- 17.17.21. image:jenkins/jenkins
- 17.17.22. image:ubuntu
- 17.17.23. image:node
- 17.17.24. image:golang
- 17.17.25. image:vimagick/scrapyd
- 17.17.26. image:yandex/clickhouse-server
- 17.17.27. image:ngrok/ngrok
- 18. Lando
- 19. Jenkins
- 20. Go
- 20.1. Basics, Command
- 20.2. Syntax, Program Flow
- 20.3. Standard Library of packages
- 20.4. Custom package, Third Party Package
- 20.5. Types
- 20.6. Concurrency
- 20.7. Structure
- 20.8. Shrink compiled binary file size
- 20.9. Project
- 20.10. To Learn
- 21. Vagrant
- 22. Apache
- 22.1. Basics
- 22.2. Config loading
- 22.3. Modules
- 22.4. Directive Context apache:directive context
- 22.5. Core Directives
- 22.6. LogFormat
- 22.7. Basic authentication
- 22.8. Production Setup apache:production
- 22.9. Force redirect from non www to www
- 22.10. Force redirect to https .htaccess
- 22.11. Restrict access to a file or sub folder
- 22.12. WebP
- 22.13. SSL apache:ssl
- 22.14. Redirect to HTTPS apache:https
- 23. Nginx
- 23.1. Install on Ubuntu 16.04
- 23.2. /etc/nginx /var/log/nginx
- 23.3. Installed modules
- 23.4. Config
- 23.5. server Block nginx:b:server
- 23.6. location block
- 23.7. proxy_pass directive
- 23.8. Rewrite directive
- 23.9. listen Directive
- 23.10. fastcgi directives - Module ngx_http_fastcgi_module
- 23.11. add_header
- 23.12. Server name redirect
- 23.13. error_log nginx:d:error_log
- 23.14. Core Module Directives
- 23.15. log_format and access_log nginx:d:log_format
- 23.16. Let's Encrypt SSL nginx:ssl letsencrypt:certbot
- 23.16.1. Challenges
- 23.16.2. certbot nginx apache plugins
- 23.16.3. manual plugin
- 23.16.4. certbot standalone plugin
- 23.16.5. webroot plugin
--webroot-wletsencrypt:certbot:webroot - 23.16.6. manage and renew certificates
- 23.16.7. A certificate nginx:ssl:example
- 23.16.8. Windows IIS
- 23.16.9. Alternative ACME clients
- 23.16.10. Limitation
- 23.16.11. SSL config nginx:ssl:config
- 23.17. nginxconfig.io
- 23.18. deny nginx:d:deny
- 23.19. Redirect a website to another website
- 24. Git
- 24.1. Show config
- 24.2. Alias
- 24.3. Working Tree
- 24.4. Archive
- 24.5. diff
- 24.6. Patch
- 24.7. Show file names only between 2 commits
- 24.8. Stash, Reset, Clean
- 24.9. Checkout vs Reset vs Revert
- 24.10. Checkout a folder from another branch to current branch
- 24.11. Commit
- 24.12. Branch
- 24.12.1. Current branch info
- 24.12.2. Duplicate a remote branch as a new local branch
- 24.12.3. Duplicate current branch with uncommited changes and switch to it
- 24.12.4. Duplicate current branch to a commit and switch
- 24.12.5. Delete branch
- 24.12.6. Rename branch
- 24.12.7. Find branches which contain a commit
- 24.12.8. Commit difference between 2 branches
- 24.12.9. File difference between 2 branches
- 24.12.10. Find authors of all remote/local branches and tags
- 24.12.11. Make current branch exactly as another branch
- 24.12.12. Force push to remote branch
- 24.12.13. Find most recent ancestor of 2 branches. See git:merge-base
- 24.12.14. Revert to common ancestor
- 24.12.15. Find branches merged into a branch
- 24.12.16. Pull from remote/master to master branch which is not the current branch
- 24.12.17. Orphan Branch with no parent
- 24.12.18. git:upstream
- 24.13. Merge
- 24.14. merge-base
- 24.15. Rebase
- 24.15.1. Rebase backup and revert
- 24.15.2. Rebase master onto feature branch
- 24.15.3. Rebase master onto a feature branch that has merges from master
- 24.15.4. Rebase a base feature branch onto another feature branch
- 24.15.5. Interactive: Commands p(pick), s(quash), e(edit)
- 24.15.6. Branch preference
- 24.15.7. Squash commits into one commit
- 24.15.8. Reorder commits
- 24.15.9. Split a commit
- 24.16. Cherry pick
- 24.17. Show child commits for merge commit
- 24.18. Fork
- 24.19. Symlink
- 24.20. Remove tracking for commited files
- 24.21. Remove a commited file from file but leave in filesystem
- 24.22. Remove large files and folders
- 24.23. Remove one file from all remote branches
- 24.24. log - Show Commit History
- 24.25. Ignore, Local/Global Ignore, Show Ignored Files
- 24.26. Customize Settings per folder
- 24.27. Git Windows
- 24.28. Credential Manager, Multiple GitHub accounts and https
- 24.29. clone
- 24.30. Track local branch with a remote branch
- 24.31. List all remote repo
- 24.32. Maintenance, Repo Size git:repo size
- 24.33. Number of tracked files
- 24.34. GitHub Push to New Repo with local repo
- 24.35. Add a remote, remove a remote and its branches, change upstream
- 24.36. Push to 2 remotes
- 24.37. Tag
- 24.38. Submodule
- 24.39. Git-flow
- 24.40. GitHub
- 24.41. GitLab
- 24.42. SSH Key Fingerprint
- 24.43. Troubleshooting
- 25. Web Application Firewall (WAF)
- 26. SVN Subversion
- 27. BitBucket
- 28. Pantheon
- 28.1. File size
- 28.2. Timeout
- 28.3. Terminus
- 28.4. Rsync pantheon:rsync
- 28.5. pantheon.yml
- 28.6. Drush Pantheon Drush
- 28.7. WP CLI WP CLI Pantheon
- 28.8. Core Update pantheon:drupal:core
- 28.9. Drupal Cron
- 28.10. Determine Environment pantheon:environment
- 28.11. New Relic pantheon:new relic
- 28.12. Cookie pantheon:cookie
- 28.13. Domain pantheon:domain
- 28.14. Global CDN
- 28.15. HTTPS
- 28.16. Git pantheon:git
- 28.17. Logs
- 28.18. Nginx pantheon:nginx
- 28.19. Varnish - Edge Caching pantheon:varnish
- 28.20. Object Cache - Redis
- 28.21. Clear Cache
- 28.22. Lando push to Live lando:pantheon:live
- 28.23. Quicksilver Platform Hooks
- 29. Redis
- 30. AWS
- 31. Database
- 31.1. Basics
- 31.2. Access
- 31.3. MySQL
- 31.3.1. Version, Settings, Global variables
- 31.3.1.1. Get settings, global variables, session variable
- 31.3.1.2. Config: mysqld
- 31.3.1.3. MySQL Mode (sql_mode)
- 31.3.1.4. Allow remote access to user root
- 31.3.1.5. Minimum Config
- 31.3.1.6. charset, collation mysql:collation
- 31.3.1.7. Memory, InnoDB Buffer, packet
- 31.3.1.8. Last update time for a table in a database
- 31.3.1.9. Query Log
- 31.3.1.10. Threads
- 31.3.1.11. sql_mode
- 31.3.1.12. Optimization
- 31.3.2. mysql Client
- 31.3.3. SSH Tunneling
- 31.3.4. Install
- 31.3.5. MySQL 8.0
- 31.3.6. Data Types
- 31.3.7. Index, Composite Index
- 31.3.8. Common queries, functions, operators
- 31.3.8.1. Reuse SELECT alias
- 31.3.8.2. Select all columns except some columns
- 31.3.8.3.
IN - 31.3.8.4.
JOIN - 31.3.8.5.
GROUP BYandHAVING - 31.3.8.6.
COUNT - 31.3.8.7. REGEXP mysql:regex
- 31.3.8.8. DELETE
- 31.3.8.9.
INSERT - 31.3.8.10.
UPDATE - 31.3.8.11.
LIMIT,OFFSET - 31.3.8.12.
EXISTS - 31.3.8.13.
UNION - 31.3.8.14.
CASE,IF(expr1,expr2,expr3),IFNULL(expr1,expr2),NULLIF(expr1,expr2) - 31.3.8.15.
DISTINCT - 31.3.8.16. Math functions
- 31.3.8.17. Bitwise
- 31.3.8.18. String functions mysql:string:functions
- 31.3.8.19. Comparison functions and Operators
- 31.3.8.20. Transaction
- 31.3.8.21. Rows to columns
- 31.3.9. Admin queries
- 31.3.9.1. CREATE TABLE IF NOT EXITS
- 31.3.9.2. ALTER TABLE
- 31.3.9.3. Duplicate a table
- 31.3.9.4. Table Status, Table Settings, Table Definition
- 31.3.9.5. Routines
- 31.3.9.6. Deadlock
- 31.3.9.7. Transpose query results
- 31.3.9.8. Table columns
- 31.3.9.9.
EXPLAIN - 31.3.9.10.
SHOW FULL PROCESSLIST - 31.3.9.11. SIGNAL / RESIGNAL throw error or warning
- 31.3.9.12. Handler
- 31.3.9.13. Profiling
- 31.3.9.14. Row size and number of columns
- 31.3.10. User-Defined Variables
- 31.3.11. Routines, Procedures, User Defined Function
- 31.3.12. SQL Fiddle
- 31.3.13. MySQLTutorial.org
- 31.3.14. Dump, Restore
- 31.3.15. Binary Log
- 31.3.16. phpMyAdmin
- 31.3.17. Adminer db:adminer
- 31.3.18. UC: Concatenate values of a column across multiple rows about the same record
- 31.3.19. UC: Split Comma-Separated Values mysql:split list
- 31.3.20. TS: Unknown collation: utf8mb4_unicode_520_ci mysql:unknown collation
- 31.3.1. Version, Settings, Global variables
- 31.4. MariaDB
- 31.5. MSSQL
- 31.6. MongoDB
- 31.7. GraphQL
- 31.8. ClickHouse
- 32. Google Ad Manager - DFP
- 32.1. Default code - Google Publisher Tag - gpt.js
- 32.2. Why async is required
- 32.3. Competition Calculation
- 32.4. Targeting - key value, Geo
- 32.5. Exclusive Ads on Specific Page
- 32.6. Email - Non-JavaScript
- 32.7. Responsive
- 32.8. Macro in Third Party Code, Custom Code
- 32.9. Creative types
- 32.10. Examples
- 32.11. Pre-roll or Linear Video Ad
- 32.12. Native Ad, Native Styles, Custom Rendering
- 32.13. SafeFrame API
- 32.14.
googletag.Service - 32.15.
googletag.Slot - 32.16. Events
- 32.17. Google Publisher Console
- 32.18. Passback tag
- 32.19. Anti AdBlocker
- 32.20. Custom Fields
- 32.21. Reports
- 32.22. Check available inventory - Forecasting
- 32.23. Benchmarks and Standards
- 32.24. User role
- 33. XML
- 34. YAML
- 35. OpenAPI
- 36. AI
- 37. Software Programming
- 37.1. Programming Paradigm
- 37.2. Side Effect
- 37.3. DevOps
- 37.4. Visual Regression Testing
- 37.5. Security Scans
- 37.6. Performance Testing
- 37.7. Code Quality
- 37.8. Drupal
- 37.9. Machine Learning
- 37.10. Ansible
- 37.11. Licensing software:license
- 38. Tools
- 38.1. MacOS
- 38.1.1. Versions
- 38.1.2. Escape key not working
- 38.1.3. Screenshot and Screen Recording
⇧ ⌘ 5 - 38.1.4. Directories
- 38.1.5. Shortcuts
- 38.1.6. Homebrew
- 38.1.7. Display
- 38.1.8. Terminal
- 38.1.9. Safari
- 38.1.10. NFS
- 38.1.11. DNS
- 38.1.12. Photos app
- 38.1.13. Time Machine
- 38.1.14. Disk Utility
- 38.1.15. Other Apps
- 38.1.16. Logitech MX Master 3S
- 38.2. PhpStorm
- 38.2.1. Installation and 64 bit
- 38.2.2. Material Theme UI
- 38.2.3. Change config directories
- 38.2.4. Open big file
- 38.2.5. Deployment
- 38.2.6. Search, Find
- 38.2.7. Language Specifics
- 38.2.8. View
- 38.2.9. XPath
- 38.2.10. Navigate, Display, Select inside Editor
- 38.2.10.1. Edit > Paste > Paste from history (access multiple clipboards) ::
C-S-v⌘ ⇧ v - 38.2.10.2. Edit > Find
- 38.2.10.3. Edit > Find Usages
- 38.2.10.4. Show Whitespaces - Show unprintable characters (TAB)
- 38.2.10.5. Multiple cursors, move cursor, selection, navigate cursor
- 38.2.10.6. Extend selection ::
S-M-UpS-M-Down⌥ Up - 38.2.10.7. Move Caret to Line End ::
End⌘ Right⌃ e - 38.2.10.8. Move Caret to Line Start ::
Home⌘ Left⌃ a - 38.2.10.9. Move Caret to Code Block Start or end (open bracket, starting bracket) ::
C-[,C-],⌥ ⌘ [ - 38.2.10.10. Move Caret to Matching Brace ::
C-S-p⌃M - 38.2.10.11. Move Caret to Text Start (top of file) ::
⌘ Home - 38.2.10.12. Navigate > Class
C-S-t, FileC-S-r, SymbolC-M-S-n - 38.2.10.13. Navigate > Jump to navigation bar
⌘ up - 38.2.10.14. Navigate > File Structure (popup window) ::
C-F3⌘ F12 - 38.2.10.15. Navigate > Type Hierarchy
F4, Method Hierarchy, Call Hierarchy - 38.2.10.16. Navigate > Implementation(s)
C-t⌥ ⌘ B - 38.2.10.17. Navigate > Super Method (opposite of Navigagte > Implementaion(s)) ::
⌘ U - 38.2.10.18. Navigate > Declaration and Usages
F3orC-leftClickor⌘ + b - 38.2.10.19. Navigate > Select in (select opened file in other window)
⌥ + F1 - 38.2.10.20. Navigate > Column (Go to line)
⌘ + L - 38.2.10.21. Navigate > Back (move to previous place on editor) ::
⌘ [ - 38.2.10.22. Navigate > Last edit location ::
⇧ + ⌘ + Backspace - 38.2.10.23. Navigate > Next Highlighted Error (next error in an Editor tab) ::
S-F1Mac:F2⇧ F2 - 38.2.10.24. Navigate > Navigate in File > Next Change, Previous Change (Git) ::
⇧ ⌃ ⌥ Up/Down - 38.2.10.25. Navigate > Navigate in File > Previous method / Next method ::
C-S-Up,C-S-Down,⇧ ⌃ Up/Down - 38.2.10.26. View > Recent Locations
C-S-eM-left or right - 38.2.10.27. View > Parameter Info (function parameters) ::
⌘ P, ~~ - 38.2.10.28. Error Description ::
⌘ F1 - 38.2.10.29. Scroll to Centre
⌃ l - 38.2.10.30. Custom code folding regions
- 38.2.10.31. Window > Editor Tabs - Split and multiple Editor Tabs
- 38.2.10.32. Window > Active Tool Window > Maximize Tool Window
⇧ ⌘ ' - 38.2.10.33. Window > Active Tool Window > Hide all windows ::
⇧ + ⌘ + F12 - 38.2.10.34. Window > Next project window
⌘ ⌥ ` - 38.2.10.35. Open source in new window (open current file in new window) ::
S-F4⇧ F4 - 38.2.10.36. Close tab
C-F4 - 38.2.10.37. Show context menu
⌥ + Enter - 38.2.10.38. Quick Definition ::
C-S-i⌥ Spacebar - 38.2.10.39. Quick Documentation (has browser support for CSS) ::
C-S-SpaceF1 - 38.2.10.40. Goto by Reference Actions - File Path
⌥ ⌘ F12 - 38.2.10.41. Goto by Reference Actions - Jump to Navigation Bar ::
⌘ Up - 38.2.10.42. VCS Operations
⌃ + v
- 38.2.10.1. Edit > Paste > Paste from history (access multiple clipboards) ::
- 38.2.11. Difference Viewer
- 38.2.12. Project View / Project Tree
⌘ 1 - 38.2.13. Git
⌘ 9 - 38.2.14. Edits inside Editor
- 38.2.14.1. Line Comment
C-/, Block CommentC-S-/ - 38.2.14.2. To Spaces / To Tabs (convert to Tab/Spaces for indentation) :: Find Action > To Tab
- 38.2.14.3. Surround with
M-S-zorC-M-tor⌥ ⌘ t - 38.2.14.4. Code > Insert Live Template
C-M-S-j - 38.2.14.5. Postfix Template phpstorm:postfix
- 38.2.14.6. Code > Generate…
M-Insert - 38.2.14.7. Code > Folding Fold Selection / Remove region
C-. - 38.2.14.8. Code > Move Line Down / Up
⇧ ⌥ Down - 38.2.14.9. Code > Auto-Indent Lines (auto indent) ::
⌃ ⌥ i - 38.2.14.10. Code > Analyze Code > Configure Current File Analysis ::
⌥ ⇧ ⌘ h - 38.2.14.11. Code > Code Completion
- 38.2.14.12. Show Context Actions (show intention actions) ::
M-Enter⌥ Enter - 38.2.14.13. Fix Doc Comment
Find Action > fix doc comment - 38.2.14.14. Emmet
- 38.2.14.15. Start New Line Before Current
⌥ ⌘ Enter - 38.2.14.16. Edit > Join lines
C-S-j⌃ ⇧ j - 38.2.14.17. Edit > Toggle Case
⇧ ⌘ u - 38.2.14.18. Window > Editor Tabs
- 38.2.14.19. Toggle Rendered View - rendered PHPDoc
⇧ ⌥ q
- 38.2.14.1. Line Comment
- 38.2.15. Settings > Directories
- 38.2.16. Settings > Editor
- 38.2.17. Settings > Tools > Database
- 38.2.18. Settings > Advanced Settings
- 38.2.19. .editorconfig
- 38.2.20. jsconfig.json
- 38.2.21. Refactor
- 38.2.22. Help
- 38.2.23. Database
- 38.2.23.1. Setup for a project
- 38.2.23.2. Go to DLL ::
⌘ b - 38.2.23.3. Select in Database explorer ::
⌥ ⇧ b - 38.2.23.4. Full text search on a table ::
⌥ ⇧ ⌘ f - 38.2.23.5. Data Editor (table view)
- 38.2.23.6. Console
- 38.2.23.7. Return all query results
- 38.2.23.8. Icons
- 38.2.23.9. sql_mode or other session variables are not followed by global variables
- 38.2.24. Plugins
- 38.2.25. Docker support
- 38.2.26. Vue.js
- 38.2.27. WordPress Support
- 38.2.28. Xdebug
- 38.2.29. PHPUnit
- 38.2.30. Troubleshoot
- 38.3. Emacs
- 38.3.1. Starting Guides
- 38.3.2. Install & Configuration
- 38.3.3. Theme
- 38.3.4. Keybinding
- 38.3.5. MELPA packages
- 38.3.6. use-package
- 38.3.7. GitHub pkg
el-get - 38.3.8.
M-x: run a command - 38.3.9. Help ::
C-h - 38.3.10. Basics
- 38.3.11. Temporarily Change Font Size in Current Buffer
- 38.3.12. Windows
- 38.3.13. Search and replace
- 38.3.14. Region, Selection, Copy & Paste
- 38.3.15. Moving and basic editing
- 38.3.16. Headline
- 38.3.17. Code Block
- 38.3.18. Export to HTML
C-c C-e h h - 38.3.19. Symbol and Escape emacs:escape
- 38.3.20. Superscript and subscript
- 38.3.21.
TODO - 38.3.22. Sparse Tree, Search
- 38.3.23. Link
- 38.3.24. SSH remote edit files emacs:ssh:tramp
- 38.3.25. Superscript and Subscript
- 38.3.26. Include image
- 38.3.27. dot
- 38.4. VS Code
- 38.4.1. Install, Basics
- 38.4.2. Extensions
- 38.4.2.1. GitLens eamodio.gitlens
- 38.4.2.2. Live Server
- 38.4.2.3. Live Share
- 38.4.2.4. Remote SSH ms-vscode-remote.remote-ssh
- 38.4.2.5. Rest Client humao.rest-clident
- 38.4.2.6. Better Comments aaron-bond.better-comments
- 38.4.2.7. Prettier - Code formatter
- 38.4.2.8. Bracket Pair Colorizer 2
- 38.4.2.9. Python extension for Visual Studio Code vs:code:ext:my-python.python
- 38.4.2.10. Python Docstring Generator njpwerner.autodocstring
- 38.4.2.11. JavaScript (ES6) code snippets xabikos.javascriptsnippets
- 38.4.2.12. IntelliJ IDEA Keybindings
- 38.4.2.13. PHP related extensions
- 38.4.2.14. ESLint
- 38.4.2.15. npm Intellisense christian-kohler.npm-intellisense
- 38.4.2.16. Import Cost wix.vscode-import-cost
- 38.4.2.17. stylelint
- 38.4.2.18. Glitch for VS Code vs:code:ext:glitch.glitch
- 38.4.2.19. Sass syler.sass-indented
- 38.4.2.20. MJML vs:code:ext:attilabuti.vscode-mjml
- 38.4.3. View
- 38.4.4. Shortcuts
- 38.4.5. Setting
- 38.5. WinMerge
- 38.6. WinSCP
- 38.7. Regex
- 38.8. Chrome
- 38.8.1. Resources
- 38.8.2. Command Menu (DevTools > C-S P)
- 38.8.3. Toggle/detach dock side (DevTools > C-S D)
- 38.8.4. Navigate Panels (DevTools >
C [orC ]) - 38.8.5. Navigate among tabs
⌘ ⇧ a⌥ ⌘ left/right - 38.8.6. Console Panel
- 38.8.7. Block Request or Domain - Network
- 38.8.8. Get all events of an element in Console
- 38.8.9. XPath XPath
- 38.8.10. Turn on DevTools Experiment
- 38.8.11. Elements Panel
- 38.8.12. Audits Panel
- 38.8.13. Sources Panel
- 38.8.14. Extension
- 38.8.14.1. Google Tag Assistant chrome:google tag assistant
- 38.8.14.2. Postman
- 38.8.14.3. Save Page WE
- 38.8.14.4. Ignore X-Frame headers
- 38.8.14.5. Requestly
- 38.8.14.6. jQuery Audit
- 38.8.14.7. Web Server for Chrome
- 38.8.14.8. Wasp chrome:wasp
- 38.8.14.9. Hosts file
- 38.8.14.10. ChroPath
- 38.8.14.11. Nimbus Screenshot & Screen Video Recorder chrome:extension:nimbus-screenshot-screen
- 38.8.14.12. React Developer Tools
- 38.8.15. Network
- 38.8.16. Change User Agent
- 38.8.17. Which DOM elements are repainted
- 38.8.18. Performance
- 38.8.19. Clear cache for one domain
- 38.8.20. Refresh Favicon
- 38.8.21. Flush DNS Caches in Chrome
- 38.8.22. Flash Couldn't Load Plugin
- 38.8.23. Allow js:xmlhttprequest for local files
- 38.8.24. Allow CORS locally
- 38.8.25. Remote Debug Android Devices
- 38.8.26. Debug
- 38.8.27. Command line
- 38.8.28. Security chrome://settings/security
- 38.9. Firefox
- 38.10. BrowserStack
- 38.11. Device Market share, Metrics
- 38.12. Windows
- 38.12.1. Win Shortcuts
- 38.12.2. File Explorer
- 38.12.3. Win 7
- 38.12.4. Win 10
- 38.12.4.1. Boot to Safe Mode with Boot Options Menu
- 38.12.4.2. Install a printer loop in "Type a printer name"
- 38.12.4.3. Windows version
- 38.12.4.4. Upgrade from Windows 7
- 38.12.4.5. Ubuntu Bash on Windows
- 38.12.4.6. WSL 2
- 38.12.4.7. Win 10 Build Update History
- 38.12.4.8. Startup Programs and Change HOMEDRIVE, HOMEPATH
- 38.12.4.9. Remove a keyboard
- 38.12.4.10. Move a window
- 38.12.5. Win Server 2003
- 38.12.6. PowerSehll
- 38.12.7. Wireless hotspot
- 38.12.8. SOCKS Proxy
- 38.12.9. Windows Virtual PC
- 38.12.10. Find physical path by shared path
- 38.12.11. Process Monitor procmon.exe
- 38.12.12. Dosbox
- 38.12.13. Batch Script
- 38.12.14. FART windows:FART
- 38.12.15. Windows Credential Store Panel Windows Credential Store Panel
- 38.12.16. MAC address
- 38.12.17. Meter an Ethernet Connection
- 38.12.18. TS: "Installation Directory must be on a local hard drive"
- 38.12.19. TS: Win 10 random freezes
- 38.13. Office 365
- 38.14. Cmder
- 38.15. Reflector, ILSpy and Reflexil
- 38.16. Project Management
- 38.17. Slack
- 38.18. Image
- 38.19. cPanel WHM
- 38.19.1. WHM Auto Backup
- 38.19.2. WHM/cPanel DNS Record Files
- 38.19.3. WHM Add IP to Firewall - Permit SSH Access
- 38.19.4. WHM List of Accounts
- 38.19.5. WHM Security Advisor
- 38.19.6. WHM Access
- 38.19.7. SSH access
- 38.19.8. autossl:whm
- 38.19.9. Remote MySQL connection - cPanel
- 38.19.10. Access Log - cPanel
- 38.19.11. Root domain A record change - cPanel
- 38.20. Web Hosting
- 38.21. Tech Stack
- 38.22. Website Status
- 38.23. Website Security, Testing
- 38.24. Webhook
- 38.25. Send password
- 38.26. RESTful API
- 38.27. Screencast + Audio, FFmpeg
- 38.28. Sketch Diagrams
- 38.29. Video
- 38.30. Audio
- 38.31. Google
- 38.31.1. Google Public DNS
- 38.31.2. Google S2
- 38.31.3. Google reCaptcha
- 38.31.4. Google Search - Advanced
- 38.31.5. Google Ads Personalization Setting
- 38.31.6. Google Partners
- 38.31.7. Gmail
- 38.31.8. Google Sheets
- 38.31.9. Google Maps
- 38.31.10. Google Correlate
- 38.31.11. Google Planning Tools
- 38.31.12. Tools for Web Developers
- 38.31.12.1. Lighthouse in DevTools chrome:lighthouse
- 38.31.12.2. Mobile-Friendly Test WebPageTest.org provided by Google
- 38.31.12.3. PageSpeed google:pagespeed
- 38.31.12.4. Fetch as Google google:search console
- 38.31.12.5. Rich Snippet Testing Tool, Structured Data Testing Tool google:json-ld
- 38.31.12.6. Google Site Status (security) google:site status
- 38.31.12.7. Install with headless chromium
- 38.31.13. Google Experts
- 38.32. Translation
- 38.33. HTML Minimizer, document.write
- 38.34. Data file validator and visualizer
- 38.35. File Transfer
- 38.36. CloudConvert.com
- 38.37. Online App Publishing
- 38.38. Email Service and Email Marketing
- 38.39. Payment Gateway
- 38.40. Market Research market research:firm
- 38.41. W3C
- 38.42. Adobe
- 38.42.1. Acrobat Pro DC
- 38.42.2. Creative Cloud
- 38.42.3. Photoshop
- 38.42.4. Illustrator
- 38.42.4.1. Setting
- 38.42.4.2. Panels, Workspace, Artboard
- 38.42.4.3. Hand, Zoom, Screen mode, View, Rulers, Guides
- 38.42.4.4. Repeat the previous action
- 38.42.4.5. Move, Select
- 38.42.4.6. Pencil and Brush tools
- 38.42.4.7. Basic Shapes
- 38.42.4.8. Drawing Modes
- 38.42.4.9. Transform Object
- 38.42.4.10. Fill and Stroke
- 38.42.4.11. Color, Swatch
- 38.42.4.12. Appearance
- 38.42.4.13. Complex Shapes, Compound Path, Pathfinder, Shape Builder
- 38.42.4.14. Eraser
- 38.42.4.15. Pen Tool
- 38.42.4.16. Type Tool
- 38.42.4.17. Copy from InDesign
- 38.42.4.18. Resize canvas to selection
- 38.42.4.19. Raster Image
- 38.42.4.20. Crop image
- 38.42.4.21. Export Assets
- 38.42.4.22. Export SVG
- 38.42.4.23. Scripts
- 38.42.5. Color CC
- 38.43. Digital Edition
- 38.44. Publication
- 38.45. CRM
- 38.46. Automations
- 38.47. Form Service
- 38.48. Survey Service
- 38.49. Merchandise
- 38.50. HR
- 38.51. Training
- 38.52. Work Ethics
- 38.53. TeamViewer
- 38.54. Font
- 38.54.1. Abbreviations Foundary
- 38.54.2. TS: Not a valid font file Windows
- 38.54.3. Line height
- 38.54.4. Open source alternative, identify font
- 38.54.5. Variable Fonts
- 38.54.6. Windows Font List
- 38.54.7. iOS Font List
- 38.54.8. Extensis
- 38.54.9. Adobe Fonts former Adobe TypeKit (AF)
- 38.54.10. Adobe Edge Web Fonts (AEWF)
- 38.54.11. Google Font (GF)
- 38.54.12. FontForge
- 38.54.13. Social Media Icons wp:plugin:simple-social-icons
- 38.54.14. Monospace
- 38.54.15. Material Font, GF
- 38.54.16. Poppin, GF
- 38.54.17. Museo, GF
- 38.54.18. Roboto Slab, GF
- 38.54.19. IcoMoon
- 38.54.20. Fontello
- 38.55. Digital Agency
- 38.1. MacOS
- 39. Industry Compliance, Standards
- 40. TODOs
- 40.1. TODO https://www.gitbook.com/
- 40.2. TODO https://blog.pragmaticengineer.com/my-reading-list/
- 40.3. PHP catch fatal error then close MySQL transaction
- 40.4. https://www.pdftron.com/?utm_source=google&utm_medium=cpc&utm_campaign=branded&utm_content=pdftron&utm_term=pdftron&gclid=Cj0KCQiA9P__BRC0ARIsAEZ6irgMeBwmMtR70-tDS52EFyCLFEQ1wtatx-kF-cxskHtZTnHGbkQ-JnYaAtgvEALw_wcB
- 40.5. https://www.youtube.com/playlist?list=PLNYkxOF6rcIDzLmWaDwfHVZJl1Q5RFgOR
- 40.6. dialpad.com, aloware.com
- 40.7. ClickHouse.tech
- 40.8. DONE MySQL sql mode
- 40.9. TODO Vagrant on Hyper-V
- 40.10. TODO CSS Nesting
- 40.11. TODO Inspectlet.com
- 40.12. TODO QUnit
- 40.13. TODO Learn Continuous Integration "Pantheon Build Drupal with Composer on Travis"
- 40.14. TODO Learn tool Zeplin
- 40.15. TODO Lynda Web Project Workflows with Gulp.js, Git, and Browserify
- 40.16. DONE Lynda Agile Project Management
- 40.17. TODO Lynda Firebase, Travis CI, Heroku
- 40.18. TODO JavaScript HTML5 Animation Framework: GSAP or Tween https://greensock.com/
- 40.19. TODO Lynda: MVC Frameworks for Building PHP Web Applications with Drew Falkman
- 40.20. TODO Book: Code Complete (2nd Edition, Microsoft Press)
- 40.21. TODO custom-elements-everywhere.com, williams sonoma
- 40.22. TODO gridbyexample.com
- 40.23. TODO BrowserStack and BeHat
- 40.24. TODO SearchKings.ca
- 40.25. TODO Agency DriveDigital.ca
- 40.26. TODO Front End Design
- 40.27. TODO Web agencies backed by WP Engine
- 40.28. TODO https://developers.google.com/training/
- 40.29. TODO https://grow.google/
- 40.30. TODO https://www.thinkwithgoogle.com/intl/en-ca/
- 40.31. TODO https://experiments.withgoogle.com/chrome
- 40.32. TODO https://www.sitepoint.com/nuxt-js-universal-vue-js/
- 40.33. TODO https://generalassemb.ly/ Course on Product Management
- 40.34. TODO https://www.roberthalf.com/ Job Hunt
- 40.35. TODO https://developers.google.com/experts/ https://abrah.am/
- 40.36. TODO http://grow.io/
- 40.37. TODO https://www.ontario.ca/page/digital-government https://medium.com/ontariodigital
- 40.38. TODO Learn New Relic One
- 40.39. TODO Learn https://langserver.org
- 40.40. TODO Python Standard Library Essential Training
- 40.41. TODO Learn Rust to develop API, Restful Microservices
- 40.42. TODO OMCP certification
- 40.43. TODO Certified ScrumMaster (CSM) vs Professional Scrum Master (PSM)
- 40.44. TODO Book: Nonviolent Communication: A Language of Life
- 40.45. DONE Lynda Behance.net
- 40.46. DONE Lynda Illustrator CC for Web Design: Core Concepts, Aesthetics, Image Optimization, Wireframing, SVG
- 40.47. TODO Semantic-UI.com, Slick JS, Typed.js
- 41. Random
1. PHP
1.1. Installation
1.1.1. Debian
- Supported versions
- https://www.digitalocean.com/community/tutorials/how-to-run-multiple-php-versions-on-one-server-using-apache-and-php-fpm-on-ubuntu-18-04
Install multiple versions of PHP
# install an apt repo which allows you to switch PHP versions # installed on Ubuntu 18.04 sudo apt install software-properties-common sudo add-apt-repository ppa:ondrej/php sudo apt-get update # install PHP for Apache (install Apache and everything?) sudo apt install php7.3 # install PHP as FPM (one type of FastCGI, full name FastCGI Process Manager). Needed for Nginx sudo apt install php7.3-fpm # install a specific module, use tab sudo apt install php7.3-cli # install multiple modules # sudo apt install php7.3-cli php7.3-xml php7.3-mysql sudo apt install php7.3-{cli,xml,mysql,zip,mbstring,curl} # after install, php-fpm services should be auto started sudo systemctl status php7.3-fpm # change PHP version as an Apache module only a2dismod php7.0 ; a2enmod php5.6 ; service apache2 restart # check PHP version php -v # see what alternatives of PHP update-alternatives --config php # set default PHP version for PHP CLI only sudo update-alternatives --set php /usr/bin/php7.3 # get location of php.ini, usually `/etc/php/7.3/cli/php.ini` php -i | grep "Loaded Configuration File"
1.1.2. Xdebug
apt update apt install php7.4-xdebug # find which xdebug ini file is loaded php --ini | grep xdebug # e.g. /etc/php/7.4/cli/conf.d/20-xdebug.ini # /etc/php/7.4/cli/conf.d/20-xdebug.ini # and # /etc/php/7.4/fpm/conf.d/20-xdebug.ini both are symlink to: # /etc/php/7.4/mods-available/xdebug.ini # add the following line to /etc/php/7.4/mods-available/xdebug.ini
1.1.2.1. Settings
zend_extension=xdebug.so xdebug.mode=debug,profile # xdebug.start_with_request=default means trigger for debug, yes for profile xdebug.start_with_request=trigger # php -d xdebug.profiler_enable=On script.php xdebug.idekey=docker # The internal IP of the docker host (Mac). Eq. `ipconfig getifaddr en0` xdebug.client_host=host.docker.internal # https://xdebug.org/docs/profiler # default. %p for PID. Other %s for script location # xdebug.profiler_output_name=cachegrind.out.%p # https://xdebug.org/docs/profiler ends # Xdebug v3 uses port 9003 as default # xdebug.client_port=9003 # # Xdebug introduces nesting level e.g. recursion. Default is 256 # xdebug.max_nesting_level=999999 # create a webpage that runs: # <?php # var_dump(xdebug_info()); # Or # php --ri xdebug # Xdebug version: php -v
1.1.2.2. Profiling
- Output directory
php --ri xdebug | grep xdebug.output_dir- default
/tmp
- Install an app on MacOS
brew install qcachegrind- Then open a profiling file
qcachegrind cachegrind.out.1- (no term)
- Metrics
- 100,000 * 10ns = 1ms
- the call stack in which the function (selected on the left panel) is involved
- Lower right panel
- Callees
- callees at distnace 1 (first level)
- Call graph
- significant callers and callees at different distance
- All Callees
- callees at all distance
1.1.2.3. PhpStorm set up
- Xdebug is installed inside a docker with PHP with port 80 exposed and mapped to docker host port 80
- https://dev.to/jackmiras/xdebug-in-phpstorm-with-docker-2al8
- Make sure Settings > Build, Execution, Deployment > Docker is connection successful
- Docker for Mac: default unix:///var/run/docker.sock
- This Docker setting is shared across projects in PhpStorm
- Create a Server on PhpStorm (A debugging server on local)
- Settings > PHP > Servers
- Host: localhost
- Port: 80
- Debugger: Xdebug
- Use path mappings
- Settings > PHP > Servers
- Add PHP Remote Debug
- On the main debug tool, click on 3 vertical dots and choose edit
- Add New Configuration > PHP Remote Debug
- Filter debug connection by IDE key: check
- Server: the server we just created above
- IDE key (session id): docker
- Filter debug connection by IDE key: check
- Click Debug button
- Extra settings
- PHP > Debug
- PHP > Debug > Xdebug
- Force break at first line when no path mapping specified: uncheck
- Force break at first line when a script is outside the project: uncheck
- PHP > Debug > Settings
- Notify if debug session was finished without being paused: uncheck
- PHP > Debug > Xdebug
- PHP > Debug
1.2. Configuration & CLI
1.2.1. CLI
- Get php configuration files location
php --ini<?php var_dump(php_ini_loaded_file()); ?>
- (no term)
- Get loaded extensions/modules
php -r "print_r(get_loaded_extensions());"-r- run PHP code without using script tags
<?..?>
- show configuration for extension xdebug
- all loaded modules
php -m | grep -e redis- see if a module is installed
- Get a php config variable
php -i | grep --color -n "memory_limit"<?php echo ini_get('memory_limit'); ?>
- All CLI options
- https://www.php.net/manual/en/features.commandline.options.php
php -i- eq. to
phpinfo() php -d xdebug.profiler_enable=On script.php- override INI settings
- Interactive shell mode
php -a, typequit
1.2.1.1. Command line arguments
php script.php value1 value2
if (isset($argc)) { for ($i = 0; $i < $argc; $i++) { echo "Argument #" . $i . " - " . $argv[$i] . "\n"; } } else { echo "argc and argv disabled\n"; } /* Argument #0 - script.php Argument #1 - value1 Argument #2 - value2 */
# assign command line arguments as $_GET php -f a.php a=1 b[]=2 b[]=3
// a.php parse_str(implode('&', array_slice($argv, 1)), $_GET); $_GET === [ 'a' => '1', 'b' => ['2', '3'], ]; // URL: ?id[gte]=2&id[]=1&id[]=3 $_GET === [ 'id' => [ 'gte' => 2, 0 => 1, 1 => 3 ], ];
// Wait for input echo "Are you sure you want to do this? Type 'yes' to continue: "; $handle = fopen ("php://stdin","r"); $line = fgets($handle); if (trim($line) != 'yes') { echo "ABORTING!\n"; exit; } fclose($handle); echo "\n"; echo "Thank you, continuing...\n";
$fieldA = (function ($value) { $result = ($_f = function ($fieldA = null) use (&$_f) { if (! intval($fieldA)) { echo "Please provide a fieldA (integer):\n"; $handle = fopen('php://stdin', 'r'); $line = intval(trim(fgets($handle))); fclose($handle); $fieldA = (! $line) ? $_f() : $line; } return intval($fieldA); })($value); echo "fieldA is set to {$result}\n"; return $result; })($argv[1] ?? null);
abc=1 php a.php
$abc = getenv('abc'); $abc = ($abc === false) ? 'not exists' : $abc;
1.2.2. Basics
Changing php.ini requires PHP restart and apache:restart or Nginx
# raw way /etc/init.d/php7.4-fpm restart # SystemD systemctl | grep 'php' systemctl restart php7.4-fpm.service
1.2.2.1. PHP Version
phpinfo(); echo phpversion(); echo PHP_VERSION;
1.2.2.2. Possible places for php.ini
- 7.x
- Ubuntu
- Apache
- /etc/php/7.x/apache2/php.ini, /etc/php/7.0/apache2/conf.d/*.ini
- CLI
- /etc/php/7.x/cli/php.ini, /etc/php/7.x/cli/conf.d/*.ini
- Debian
- Nginx (FPM)
- image:php:7.x-fpm:ini
- Ubuntu
- /usr/local/etc/php/php.ini, /usr/local/etc/php/conf.d/*.ini
- /etc/php5/apache/php.ini, /etc/php5/cli/php.ini
1.2.2.3. php.ini vs .user.ini
Put .user.ini in website root directory. Only place php.ini in main config or conf.d directory or a directory that is setup to scan php.ini files. Placing php.ini in website root directory only works for that directory but not recursively in sub directories.
1.2.2.4. Apache
- Set PHP via Apache directives e.g. httpd.conf .htaccess
- Require
AllowOverride OptionsorAllowOverride All - Cannot use PHP constants e.g. E_ALL
- Changing this does not require to reload Apache nor PHP
And then
// for value. Can be used only with PHP_INI_ALL and PHP_INI_PERDIR type directives php_value name value // for boolean. Can be used only with PHP_INI_ALL and PHP_INI_PERDIR type directives php_flag name on|off // This cannot be set in .htaccess file and cannot be overriden by .htaccess or ini_set() php_admin_value name value // This cannot be set in .htaccess file and cannot be overriden by .htaccess or ini_set() php_admin_flag name on|off
1.2.2.5. Configuration Mode
- PHP_INI_USER
- Entry can be set in user scripts (like with ini_set()) or in the Windows registry. Since PHP 5.3, entry can be set in .user.ini
- PHP_INI_PERDIR
- Entry can be set in php.ini, .htaccess, httpd.conf or .user.ini (since PHP 5.3)
- PHP_INI_SYSTEM
- Entry can be set in php.ini or httpd.conf
- PHP_INI_ALL
- Entry can be set anywhere
1.2.2.6. Directive Value
Value can be
- a string
Empty string
foo = ; sets foo to an empty string foo = None ; sets foo to an empty string foo = "None" ; sets foo to the string 'None'
- a number
- e.g.
E_ALLorM_PI - On, Off, True, False, Yes, No and None
- an expression
- e.g.
E_ALL&~E_NOTICE - a quoted string ("bar")
- e.g.
${foo} - Expressions in the INI file are limited to bitwise operators and parentheses:
| bitwise OR
define('A',1); define('B',1<<1); define('C',1<<2); $a = A; $b = $a | B; $c = $b | C; $c2 = A; $c2 |= B; $c2 |= C; $d = A | B | C; $c === $d; $c === $c2;
- ^ bitwise XOR
- & bitwise AND
- ~ bitwise NOT
- ! boolean NOT
- e.g.
- Boolean
- True
- use
1,On,TrueorYes - False
- use
0,Off,FalseorNo
1.2.3. [PHP]
1.2.3.1. magic quotes
Ignore. They are deprecated. Should be Off at all times
; Magic quotes for incoming GET/POST/Cookie data. magic_quotes_gpc = Off ; Magic quotes for runtime-generated data, e.g. data from SQL, from exec(), etc. magic_quotes_runtime = Off ; Use Sybase-style magic quotes (escape ' with '' instead of \'). magic_quotes_sybase = Off
1.2.3.2. allow_url_fopen
Default is 1 to enable. If disabled, file_get_contents is not possible. This setting can only be set PHP_INI_SYSTEM
1.2.3.3. auto_prepend_file
// default auto_prepend_file = auto_prepend_file = /var/www/prepend.php if (isset($_SERVER['HTTP_X_FORWARDED_FOR'])) { $_SERVER['REMOTE_ADDR'] = $_SERVER['HTTP_X_FORWARDED_FOR']; }
1.2.3.5. session
- http://us3.php.net/manual/en/session.configuration.php
- php:session
- Most of them are PHP_INI_ALL
1.2.3.6. extension_dir php.ini:extension_dir
1.2.3.7. max_execution_time php.ini:max_execution_time
- PHP_INI_ALL, default 30s
set_time_limit( int $seconds )can extend the limit by$secondsfor one PHP script file
1.2.3.8. include_path
- See 1.9.3
- use
:to separateinclude_path = ".:/usr/share/php" - use
;to separateinclude_path = ".;c:\php\includes" - get_include_path()
- set_include_path()
1.2.4. Enable extension php.ini:extension_dir
php.ini can also enable module. e.g. to enable module intl. Use php.ini:extension_dir
; extension=modulename ; For example: extension=mysqli extension=/path/to/extension/mysqli.so ; xdebug zend_extension = /path/to/extension/xdebug.so
http://php.net/manual/en/extensions.alphabetical.php https://pecl.php.net/packages.php
1.2.5. prod
; Show all errors, except for notices and coding standards warnings. error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT display_errors=Off log_errors=On ; error_log not set go to Apache error log ; Refer to apache:core:serversignature expose_php = Off memory_limit = 128M post_max_size = 8M max_execution_time = 30 max_input_time = 60 ; default no functions are disabled ; disable_functions = show_source, exec, shell_exec, system, passthru, proc_open, popen file_uploads = On max_file_uploads = 20 upload_max_filesize = 2M allow_url_fopen = On ; Whether to allow `include`/`require` to open URLs (like http:// or ftp://) as files. allow_url_include = Off ; php:session ; in seconds session.cookie_lifetime = 0 session.cookie_secure = 1 ; 0 if not using SSL session.cookie_httponly = 1 session.use_only_cookies = 1 ; ID can't come from GET or POST ; session.entropy_file = "/dev/urandom" ; add more randomness bits session.gc_maxlifetime=1440 ; seconds, default. Unused PHP session will be kept alive for 24 miutes.
1.2.6. dev
1.2.7. Mail
http://php.net/manual/en/mail.configuration.php
[mail function] ; For Win32 only. ; http://php.net/smtp SMTP = localhost ; http://php.net/smtp-port smtp_port = 25 ; For Win32 only. ; http://php.net/sendmail-from ;sendmail_from = me@example.com ; For Unix only. You may supply arguments as well (default: "sendmail -t -i"). ; http://php.net/sendmail-path ;sendmail_path = ; Force the addition of the specified parameters to be passed as extra parameters ; to the sendmail binary. These parameters will always replace the value of ; the 5th parameter to mail(). ;mail.force_extra_parameters = ; Add X-PHP-Originating-Script: that will include uid of the script followed by the filename mail.add_x_header = On ; The path to a log file that will log all mail() calls. Log entries include ; the full path of the script, line number, To address and headers. ;mail.log = ; e.g. change to ; mail.log = /var/log/phpmaillog ; Log mail to syslog (Event Log on Windows). ;mail.log = syslog
Enable mail.log, make sure /var/log/phpmaillog has www-data:www-data as the apache
You may need to change the sendmail_path to a php wrapper instead of just adding mail.log location http://www.matteomattei.com/how-to-log-email-sent-from-php-through-mail-function/ https://www.howtoforge.com/how-to-log-emails-sent-with-phps-mail-function-to-detect-form-spam
1.2.8. Error log
- d7:show error
- 2.5
- wp:show error
- Runtime configuration for error handling
Sample
ini_set('display_errors', 1); ini_set('display_startup_errors', 1); error_reporting(E_ALL);
display_errors,display_startup_errors- PHP_INI_ALL. Default 1
- PROD always uses
Offor0
log_errors- PHP_INI_ALL. default
0. prod needs to log error instead of display_errors. If true, either to server's system log or error_log
- PHP_INI_ALL. default
log_errors_max_len- PHP_INI_ALL. default 1024 (bytes). 0 to no limit
error_log- PHP_INI_ALL. default NULL or empty. place to store error log. If not set, errors are sent to SAPI error logger e.g. Apache error log or stderr in CLI
- use whatever Linux syslog is using
error_reporting- PHP_INI_ALL
- https://www.php.net/manual/en/errorfunc.constants.php
- show all except deprecated and notice
- eq. to
E_ALL ^ (E_DEPRECATED | E_NOTICE)
- eq. to
- eq.
E_ALL & ~E_DEPRECATED Temporarily set
$oldErrorReporting = error_reporting(); error_reporting( $oldErrorReporting & ~E_WARNING & ~E_NOTICE ); // ... some code that threw warning and notice before, now won't error_reporting( $oldErrorReporting );
- Refer to php:@
- Whenever an error of any level is generated, it is formatted and ready to output to browsers. Only just before it, error_reporting setting is checked. This applies to php:@ as well
- https://stackoverflow.com/a/1869185/2196360
- 10k errors of notice level, with error_reporting and display_errors enabled
- 5k ms
- same, but with display_errors disabled
- 136 ms
- same, but with error_reporting disabled too
- 118 ms
- no error of notice level
- 20 ms
Example
// Rule of thumb, always set a valid value with the right type for a key/value if you want to manipute later // Save you a lot of work to check, validate and type conversion hassles down the road $x = []; $a = $x['abc'] + 0; // notice undefined index $x['abc'] = ''; $a = $x['abc'] + 0; // fatal error for array // warning non-numeric for string // no warning for null $a = intval($x['abc'] ?? 0); // PHP 7 using null coalescing operator. good! $a = floatval($x['abc'] ?? 0); // PHP 7. good! $a = (array)($x['abc'] ?? null); $a = intval(isset($x['abc']) ? $x['abc'] : 0); // PHP < 7. good! $a = floatval(isset($x['abc']) ? $x['abc'] : 0); // PHP < 7. good! // Use this function to have a warning/error proofing and the result could be int or float! // Use this also for PHP 7+ when the result type cannot not be predefined function toNumber($valueOrKey, $array = null, $default = 0) { $val = $valueOrKey; if (is_array($array) && isset($array[$val])) { $val = $array[$val]; } if (is_numeric($val)) { return $val + $default; } return $default; } var_dump($test = toNumber(null), $test === 0); var_dump($test = toNumber(''), $test === 0); var_dump($test = toNumber(array()), $test === 0); var_dump($test = toNumber(array(1, 2)), $test === 0); $x = array('a' => 123); var_dump($test = toNumber('keyNotExists', $x), $test === 0); var_dump($test = toNumber('a', $x), $test === 123); function arrayValue($key, $array, $default = null) { return isset($array[$key]) ? $array[$key] : $default; } $a = arrayValue('abc', $x); $a = isset($x['abc']) ? $x['abc'] : null; // $a = $x['abc']; if (isset($x['abc']) ? $x['abc'] : null) {} if (arrayValue('abc', $x)) {} // if (array_key_exists('abc', $x) && $x['abc']) {} //if ($x['abc']) {} if (is_numeric(arrayValue('abc', $x))) {} // if (isset($x['abc']) && is_numeric($x['abc'])) {} // if (is_numeric($x['abc'])) {}
1.2.9. Trigger error
Syntax
trigger_error ( string $error_msg [, int $error_type = E_USER_NOTICE ] ) : bool
- set_error_handler
1.2.10. Custom error/warning/notice hanlder
https://www.php.net/manual/en/function.set-error-handler.php
// set_error_handler returns the previous handler // Get error handler set_error_handler($handler = set_error_handler('var_dump')); var_dump('error handler', $handler); // NULL | string | array(2) | Closure restore_error_handler(); // no parameter // Set a handler for multipl error levels $oldHandler = set_error_handler( function ($errno, $errstr, $errfile, $errline, $errcontext) { //var_dump('$errno', $errno); // int, same as error level //var_dump('$errstr', $errstr); // e.g. `Undefined index: someindexname` //var_dump('$errfile', $errfile); // absolute path to file //var_dump('$errline', $errline); // int //var_dump('$errcontext', $errcontext); // deprecated as of PHP 7.2.0 and removed in PHP 8 $ignoreFiles = array('partial/path/to/file1', 'partial/path/to/file2'); $r = false; foreach ($ignoreFiles as $ignoreFile) { if (strlen($ignoreFile) > 0 ? substr($errfile, -strlen($ignoreFile)) === $ignoreFile : true) { // file path ends with $r = true; break; } } return $r; // false: nothing will be output in this function and the default PHP error handler will be run // true: output as defined in this function and the default php error handler will not be run }, E_NOTICE | E_WARNING );
1.2.11. register_shutdown_function
- Run after script execution finishes or exit() is called
- Shutdown functions can run beyond
max_execution_time Example
register_shutdown_function('shutDownLogError'); function shutDownLogError() { $e = error_get_last(); if (isset($e['type']) && $e['type'] & (E_ERROR | E_PARSE | E_CORE_ERROR | E_COMPILE_ERROR | E_USER_ERROR)) { $errstr = $e['file'] . '(' . $e['line'] . '): ' . $e['message']; badnews_logtodb($errstr); } }
1.2.12. opcache
- Preload Laravel
- https://stitcher.io/blog/preloading-in-php-74
opcache.preload=/path/to/project/preload.phppreload.php$files = /* All files in eg. vendor/laravel */; foreach ($files as $file) { // will parse but not execute opcache_compile_file($file); // to ensure linked files are also loaded, use require_once // require_once($file); }
- Load all dependecies e.g. parent class first
- To verify, restart
php-fpmandvar_dump(opcache_get_status())
1.3. Kill
ps x # find the pid kill 1234
1.4. Constant define vs const
- Predefined constants / reserved constants
- https://www.php.net/manual/en/reserved.constants.php
- (no term)
- Magic constants
- https://www.php.net/manual/en/language.constants.magic.php
- string
- current file directory path without trailing slash unless it's the root directory
- current function name or
{closure}for anonymous function- In case of closure
get_class() . ' ' . __LINE__or__TRAIT__ . ' ' . __LINE__
- the class method name
__TRAIT__
- Predefined variables
- https://www.php.net/manual/en/language.variables.predefined.php
- Reserved variables
- https://www.php.net/manual/en/reserved.variables.php
- Superglobals
- https://www.php.net/manual/en/language.variables.superglobals.php
- (no term)
- Naming
- No hyphen!
- a-z, A-Z, and the ASCII characters from 128 through 255 (0x80-0xff)
- (no term)
const- defines constant at compile time which means:
- cannot
const FOO = 'BAR';inside a condition block - cannot define a constant using a global variable
global $A; const ABC = $A;- Previous
defineandconstcan be used in the value
- Previous
- PHP 5.6+ array can be assigned and expressions are allowed
- cannot
- Can be used inside and outside of a class
- See PHP OOP
- Can define a constant using another constant
constdefines a constant in the current namespacenamespace A\B\C; // To define the constant const FOO = 'BAR'; // eq. define('A\B\C\FOO', 'BAR');
- defines constant at compile time which means:
- (no term)
define- Defines constant at run time
- can specify whether it's case sensitive or not (although it's deprecated in PHP 7.3.0)
- Drawbacks
- Scalar only for PHP 5.x, no array. 5 can define constants as array for PHP 5.6+
PHP 7.0+ can define array
define('ABC', array('a' => 1, 'b' => 2)); var_dump(ABC['a']);
- Cannot be used inside a class
// define $GLOBALS['MIN_VALUE'] define('MIN_VALUE', '0.0'); // OUTSIDE of a class definition define('MAX_VALUE', '1.0'); // OUTSIDE of a class definition if (defined('MYGLOBALVAR') && MYGLOBALVAR !== 'abc') {} const MIN_VALUE = 0.0; const MAX_VALUE = 1.0;
1.5. Protocols
1.5.1. php://
php://stdin- read only
php://stdout- write-only
$fptr = fopen(getenv("OUTPUT_PATH"), "w"); $stdin = fopen("php://stdin", "rb"); // b means binary and has to be put at the end, t means plaintext fscanf($stdin, "%d\n", $ar_count); // total number of different numbers fscanf($stdin, "%[^\n]", $ar_temp); // numbers separated by space $ar = array_map('intval', preg_split('/ /', $ar_temp, -1, PREG_SPLIT_NO_EMPTY)); $result = simpleArraySum($ar); fwrite($fptr, $result . "\n"); fclose($stdin); fclose($fptr);
1.6. $_SERVER
- Which are safe?
- GATEWAY_INTERFACE
- SERVER_ADDR
- SERVER_SOFTWARE
- DOCUMENT_ROOT
- SERVER_ADMIN
- SERVER_SIGNATURE
- Refer to 1.37
1.6.1. Partyly safe
- HTTPS
- bool whether the request is https
- HTTP_HOST
- see below
- (no term)
- REQUEST_TIME
- REQUEST_URI
/abc/xyz.html?abc=xyzwithout hash. Raw value is encoded. Useurldecode- REQUEST_METHOD
-
$method = $_SERVER['REQUEST_METHOD']; if ('PUT' === $method) { parse_str(file_get_contents('php://input'), $_PUT); var_dump($_PUT); //$_PUT contains put fields }
- Refer to 1.2
- (no term)
- REMOTE_ADDR
- REMOTE_HOST
- relies on reverse DNS lookups and may hence be spoofed by DNS attacks against server (at that time you have bigger problems anyway)
- Its value may be a proxy, which is a simple reality of the TCP/IP protocol and nothing you can do.
- (no term)
- REMOTE_PORT
- (no term)
- SERVER_PROTOCOL
- SERVER_NAME
- see below
- SERVER_PORT
- see below
- SCRIPT_NAME
/index.php. Similar to$_SERVER['PHP_SELF']which is only for PHP- SCRIPT_FILENAME
/index.php. Maybe a relative path when the script is executed with CLIREMOTE_*- values are guaranteed to be the valid address of the client as verified by a TCP/IP handshake. Because it's the address the response will be sent to
1.6.2. HTTP_HOST and SERVER_NAME
curl -H "Host: notyourdomain.com" http://yoursite.com/- By default,
HTTP_HOSTorSERVER_NAMEisnotyourdomain.com - Server might hard set (static)
SERVER_NAMEto a real machine name e.g. Pantheon - Server might dynamically set
HTTP_HOSTbased on the request e.g. Pantheon - Attackers might trigger a lot of requests and cause your website that is cached by a proxy (Varnish, Cloudflare) to have wrong cache that has content which points to the attacker's domain. Cache poisoning
- Never display
$_SERVER['HTTP_HOST']or$_SERVER['SERVER_NAME'] - For Pantheon, set
$_SERVER['SERVER_NAME'] = $_SERVER['HTTP_HOST'];
- By default,
$config['site_url'] = 'http://' . $_SERVER['HTTP_HOST'] . '/index.php'; $config['cp_url'] = 'http://' . $_SERVER['HTTP_HOST'] . '/system/index.php'; $domain = 'example.com'; $config['site_url'] = 'http://' . $domain . '/index.php'; $config['cp_url'] = 'http://' . $domain . '/system/index.php'; $domains = array('domain.com', 'dev.domain.com', 'staging.domain.com', 'localhost'); if (in_array($_SERVER['HTTP_HOST'], $domains)) { $domain = $_SERVER['HTTP_HOST']; } else { $domain = 'localhost'; }
1.6.3. SERVER_PORT
if (isset($_ENV['PANTHEON_ENVIRONMENT'])) { if (isset($_SERVER['HTTP_USER_AGENT_HTTPS']) && $_SERVER['HTTP_USER_AGENT_HTTPS'] === 'ON') { $_SERVER['SERVER_PORT'] = 443; } else { $_SERVER['SERVER_PORT'] = 80; } }
1.7. $_FILES POST method uploads
- http://www.php.net/manual/en/features.file-upload.post-method.php
<input type="file" name="userfile" >- If no name attribute, then it's
file, which means$_FILES['file']
- If no name attribute, then it's
$_FILES['userfile']['name']- The original name of the file on the client machine
- File extension e.g.
jpg$extension = strtolower(pathinfo($_FILES['userfile']['name'], PATHINFO_EXTENSION));
$_FILES['userfile']['type']- The mime type of the file, if the browser provided this information. An example would be "image/gif". This mime type is however not checked on the PHP side and therefore don't take its value for granted
$_FILES['userfile']['size']- size in bytes, of the uploaded file
$_FILES['userfile']['tmp_name']- Temporary filename of the file in which the uploaded file was stored on the server
$_FILES['userfile']['error']- Error code associated with this file upload. No error is 0
POST as array
<form action="" method="post" enctype="multipart/form-data"> <div>Pictures: <input type="file" name="pictures[]" /> <input type="file" name="pictures[]" /> <input type="file" name="pictures[]" /> <input type="submit" value="Send" /> </div> </form> <?php foreach ($_FILES["pictures"]["error"] as $key => $error) { if ($error == UPLOAD_ERR_OK) { $tmp_name = $_FILES["pictures"]["tmp_name"][$key]; // basename() may prevent filesystem traversal attacks; // further validation/sanitation of the filename may be appropriate $name = basename($_FILES["pictures"]["name"][$key]); move_uploaded_file($tmp_name, "data/$name"); } }
1.8. $_COOKIE values may also exist in $_REQUEST
1.9. Filesystem
1.9.1. Delete file
Use absolute path. See dirname
$files = [ './first.jpg', './second.jpg', './third.jpg' ]; foreach ($files as $file) { if (file_exists($file)) { unlink($file); } else { // File not found. } }
1.9.2. File Modification time
filemtime(__DIR__. "/wp-content/themes/a.css")) // int
1.9.3. Include file __DIR__, dirname(__FILE__)
Change current working directory
echo getcwd() . "\n"; // /var/www/mysite/test chdir(__DIR__ . '/../app/'); echo getcwd() . "\n"; // /var/www/mysite/app
An absolute path is defined
include __DIR__ . "/../abc."; // PHP version < 5 include dirname(__FILE__) . "/../../../../../wp-content/themes/a-theme/includes/thank-you.php"; foreach ( glob( dirname( __FILE__ ) . '/abc/*.php' ) as $file ) { include $file; }
start with
./or../cd /path/to/project # this is the entrypoint which is the result of getcwd(), current working directory php run/start.php # this is not the entrypoint! the start.php file path
include './connect.php'; // getcwd() . 'connect.php', which is /path/to/project/connect.php // if not found, return error // same applies to `include '../connect.php'`
a.phporinc/a.phpcd /path/to/project # this is the entrypoint php run/start.php
/path/to/project/run/start.php
include 'connect.php';
Try these in order
- default include paths have 2 directories
.and/usr/share/php.- stands for the entrypoint or current working directory
getcwd(), which is/path/to/projectso the file included is/path/to/project/connect.php. Which is the same behavior as relative path /usr/share/php/usr/share/php/a.php- If the file isn't found in 1.2.3.8
- (no term)
- the calling script's own directory
- (no term)
- the current working directory which is the same as
.in 1.2.3.8
- Entrypoint
- Command line
- the cd path which is
/path/to/projectin this case - Apache/Nginx
always the path to the nearest directory
- http://localhost/phpSandbox/fileinclude.php entry point is
/path/to/project/phpSandbox/- http://localhost/phpSandbox/inc/fileinclude.php entry point is
/path/to/project/phpSandbox/inc/
- cd into the directory which has the script file and run it
- Summary
- If the file to be included is in the same or deeper directory of the calling script's own directory, use bare file name or absolute path
- If the file to be included is in the parent directory of the calling script's own directory
- Try to avoid this happening
- Use absolute path
- When relative path is used be careful about different current working directory
1.9.4. file_put_contents
- Identical to calling fopen(), fwrite(), fclose() successively to write data to a file
Syntax
file_put_contents( string $filename, mixed $data, int $flags = 0, ?resource $context = null ): int|false
- Parameters
- filename
- If it does not exist, the file will be created unless
FILE_APPENDflag is set - data
- a string, an array or a stream resource
- (no term)
- flags
FILE_USE_INCLUDE_PATHFILE_APPENDLOCK_EX
- number of bytes or false on failure
file_put_contents('path/to/file', '');
1.10. Module, Extension
1.10.1. bz2
1.10.2. bcmath
1.10.3. calendar
1.10.4. enchant
1.10.5. exif
1.10.6. gd
- XPM Suport
- libXpm
- (no term)
- php:gd
1.10.7. gettext
1.10.8. intl
require debian pkg libicu-dev
1.10.9. memcached
- Port 11211
Install
sudo apt install memcached php-memcached -y sudo service apche2 restart # test echo "stats" | netcat localhost 11211 # get PHP config for memcached php -i | grep memcached # config file cat /etc/memcached.conf # restart memcached to clear cache sudo service memcached restart
- There's an old version of PHP client which has a different name
memcache. On Linux, server/daemon is stillmemcached
1.10.10. pspell
1.10.11. soap
require debian pkg libxml2-dev
1.10.12. sockets
1.10.13. tidy
1.10.14. xmlrpc
1.11. Locale
In Linux, you can run locale -a to get a list of codes
Linux return multi-letter codes. You can use 3-letter or 2-letter codes as a backup.
ISO 639 3-letter codes
1.12. Date
1.12.1. date( string $format [, int $timestamp = time() ] ) : string
- Date formats
- wp:post:post_date_gmt uses
'Y-m-d\TH:i:sP'">2019-07-16T13:46:16+00:00 - date function is related to timezone. Use
gmdate()for UTC timezone
// Y => 1999 // y => 99 or 03 // m => 01 for January // n => 1 for January // M => Jan for January // d => 01 to 31 // j => 1 to 31 // w => 0 to 6, 0 for Sunday and 6 for Saturday // H :: 24-hour format for an hour with leading zeros // G :: 24-hour format of an hour without leading zeros // h :: 12-hour format of an hour with leading zeros // g :: 12-hour format of an hour without leading zeros // i :: minutes with leading zeros // s :: seconds with leading zeros // v :: milliseconds without leading zeros. Does not work in `date()` but in `DateTime::format()` // a :: am or pm // A :: AM or PM // Use / to escape echo date('M j, Y') // Nov 1, 2020 for November 1, 2020 echo date('F j, Y'); // November 1, 2020 echo date('Ymd'); // 20190131 for Jan 31, 2019 echo date('YmdHis'); // 20190131015959 for Jan 31, 2019 01:59:59 echo date('c'); // 2019-05-24T07:11:55-07:00 echo date('y/n/j-G:i:s'); // short but unique
1.12.2. Create a custom date
1.12.2.1. Set Timezone
- Supported timezones
- https://www.php.net/manual/en/timezones.php
- (no term)
- US unique timezones
- https://stackoverflow.com/questions/4989209/list-of-us-time-zones-for-php-to-use
- Eastern Time
- America/New_York == America/Toronto
- Central Time
- America/Chicago == America/Winnipeg
- Mountain Time
- America/Denver == America/Edmonton
- Mountain Time (no DST)
- America/Phoenix
- Pacific Time
- America/Los_Angeles == America/Vancouver
- Alaska Time
- America/Anchorage
- Hawaii-Aleutian
- America/Adak
- Hawaii-Aleutian Time (no DST)
- Pacific/Honolulu
- https://stackoverflow.com/questions/4989209/list-of-us-time-zones-for-php-to-use
date_default_timezone_set('UTC'); // set the timezone to UTC date_default_timezone_set('America/Toronto'); date_default_timezone_set('America/New_York');
1.12.2.2. mktime return Unix timestamp
mktime()combined with date is very slow. Consider to use strtotime()
// hour, min, sec, month, day, year, $is_dst(daylight saving, 0 or 1) based on the current set timezone date_default_timezone_set('UTC'); // set the timezone to UTC $_u_time = mktime(11, 0, 0, 11, 22, 2016); date_default_timezone_set('America/Toronto'); $timeInTorontoTimeZone = mktime(11, 0, 0, 11, 22, 2016); $timeInUTCTimeZone = gmmktime(11, 0, 0, 11, 22, 2016); $_u_time === $timeInUTCTimeZone;
1.12.2.3. strtotime( string $time [, int $now = time() ] ) : int
- Return Unix timestamp
$_u_time = strtotime('2016-11-01'); // 20190228 format also works // ISO date string (ISO 8601). See js:time:iso // Z stands for UTC timezone $_u_time = strtotime('2011-10-05T14:48:00.000Z'); $dateStr1 = '2022-12-01 00:00:00'; $dateStr2 = '2022-12-01'; $timezone = 'America/Toronto'; var_dump(strtotime($dateStr1 . ' '. $timezone) === 1669870800); var_dump(strtotime($dateStr2 . ' '. $timezone) === 1669870800); // Compare if($start > $end) {} // difference in seconds echo $end - $start; // Format differences // convert seconds to DateTime object $dt_start = new DateTime(date('Y-m-d H:i:s', $start)); $dt_end = new DateTime(date('Y-m-d H:i:s', $end)); $interval = date_diff($dt_end, $dt_start); echo $interval->format('%R%a days'); echo $interval->y; // years // yesterday $hour = 12; $today = strtotime($hour . ':00:00'); $yesterday = strtotime('-1 day', $today); $dayBeforeYesterday = strtotime('-1 day', $yesterday); $startOfADay = strtotime('today', $anyUnixTimeStamp); $startOfADay == mktime(0, 0, 0, ...explode('-', date('n-j-Y', $anyUnixTimeStamp)) // strtotime('first day of last month') echo date('Ymd', $today).PHP_EOL; echo date('Ymd', $yesterday).PHP_EOL; echo date('Ymd', $dayBeforeYesterday).PHP_EOL;
1.12.2.4. DateTime, DateInterval
Create by using Unix timestamp (int) in seconds
$_u_time = strtotime('2016-11-01'); // 20190228 also works $a = new DateTime(); // now $a->setTimestamp($_u_time); // set it based on a Unix timestamp to current set timezone $a->getTimestamp(); // convert back to integer Unix timestamp $now = new DateTime(); $interval = $now->diff($a); echo $interval->format('%R%a days'); $yesterday = new DateTime(); // now $yesterday->sub(new DateInterval('P1D')); // subtract one day echo $yesterday->format('Ymd'); $myDate = '02111981 00:00:00'; /** @var DateTime|false $date */ $newDate = DateTime::createFromFormat('mdY H:i:s', $myDate); // Current set timezone $newDate = DateTime::createFromFormat('mdY H:i:s', $myDate, new DateTimeZone('UTC')); // At a timezone other than the current set timezone $newDateString = ($date) ? $date->format('Y-n-j') // error if $date is false : false; $newDateUnix = ($date) ? $date->getTimestamp() : null;
Convert a string format to another
// to convert a more complex string format to date // return false if it is not of that format // http://php.net/manual/en/datetime.createfromformat.php $date = DateTime::createFromFormat( 'j-M-Y', '15-Feb-2009'); $date_display = $date->format('Y-m-d');
1.12.3. Unix to Format String
$a = strtotime('20190228'); echo gmdate("Y-m-d", $a);
1.12.4. Date in Other Languages (Locale)
$_u_time = mktime (11, 0, 0, 11, 22, 2016); setlocale(LC_TIME, 'fr_FR.utf8','fra'); // try fr_FR.utf8 first, if fail, try fra $_fr_time = array(); $_fr_time[] = strftime('%A', $_u_time); // format a date/time according to locale settings $_fr_time[] = trim(strftime('%e', $_u_time)); // day of month, single digit with leading space if it's single digit.. $_fr_time[] = strftime('%B', $_u_time); $_fr_time[] = strftime('%Y', $_u_time); $_fr_time = ucwords(implode(' ', $_fr_time)); echo $_fr_time;
1.12.5. Change timezone
$timezone = date_default_timezone_get(); // UTC var_dump($timezone); // UTC var_dump(date('Y/m/d h:i:s a',time())); // e.g. 7pm date_default_timezone_set('America/Toronto'); var_dump(date('Y/m/d h:i:s a',time())); // e.g. 7pm +4 = 11pm date_default_timezone_set($timezone); // UTC var_dump(date('Y/m/d h:i:s a',time())); // e.g. 7pm
1.13. Bitwise Operator
:PROPERTIES: :ID: 3E3C7306-E75A-46ED-83DA-989FB774944D :CUSTOM_ID: php:bitwise END:
// $a << $b is to shift the bits of $a to left for $b steps (each step multiplyies by two) define('FLAG_A', 1); /// 0001 or 1<<0 define('FLAG_B', 2); /// 0010 or 1<<1 define('FLAG_C', 4); /// 0100 or 1<<2 define('FLAG_D', 8); /// 1000 or 1<<3 // $a >> $b is to shift the bits of $a to right for $b steps (each step divides by two) var_dump(8 >> 3 === 1); // 1000 to 1 // Allocated memory in megabytes: becaues 1024 * 1024 = 2^10 * 2*^10 = 2^20 bytes equals to 1 megabyte $memory = memory_get_usage(true) >> 20; // 64-bit PHP var_dump(PHP_INT_MAX === 9223372036854775807); // 62 is the maximum var_dump(1<<62 === 4611686018427387904); $combined_flags = FLAG_A | FLAG_C; // 0101 binary, which is decimal 5 var_dump(decbin($combined_flags) === '101'); var_dump(($combined_flags & FLAG_C) === FLAG_C); // FLAG_C which is 0100, which is decimal 4 // 0101 :: FLAG_A or FLAG_C // 0100 :: FLAG_C // 0100 :: result is decimal 4 if ($combined_flags & FLAG_C) { var_dump('Has FLAG_C assuming FLAG_C is 1 >> n'); } $combined_flags2 = $combined_flags | 0; var_dump( $combined_flags2 === $combined_flags); var_dump( $combined_flags ^ FLAG_C ); // XOR :: like | but not both // 0101 :: FLAG_A or FLAG_C // 0100 :: FLAG_C // 0001 :: result is decimal 1
1.14. Ternary Operator, Null Coalescing Operator, Elvis Operator, Short if
- See 8.7
// null coalescing operator (??) since PHP 7.0 // It returns its first operand if it exists and is not NULL; otherwise it returns its second operand. // eq. $foo = $bar['id'] ?? 'something'; $foo = isset( $bar['id'] ) ? $bar['id'] : 'something'; // Can be used to assign the first non null value. If all values are null, null is used $foo = $a ?? $b ?? $c; // Can be used to var_dump to debug $foo = [ 'a' => $longExpression ]; $foo = [ 'a' => var_dump('description', $longExpression = 'xxx') ?? $longExpression ]; // Elvis Operator (?: a short form of `expression1 ? expression2 : expression3`) $foo = isset($bar['id']) ?: 'something'; // eq. $foo = (isset( $bar['id'])) ? (isset($bar['id'])) : 'something'; $bar = abc(); // $bar is always set and it could be 0 or some other non-null values $foo = $bar ?: 'null'; // non-true values are converted to 'null' and other true values are kept // eq. $foo = ($bar) ? $bar : 'null'; // Elvis Operator still throws exception if there's undefined variables // Since 7.4 $a['abc'] ??= 1; // eq. $a['abc'] = $a['abc'] ?? 1; // PHP 8 null safe operator can work with methods // Instead of writing this $country = null; if ($session !== null) { $user = $session->user; if ($user !== null) { $address = $user->getAddress(); if ($address !== null) { $country = $address->country; } } } // write this $country = $session?->user?->getAddress()?->country; // PHP 7.3 we can still use `??` but it only works for properties var_dump($a->aProperty ?? 'not defined'); // This does not work // var_dump($a->aMethod() ?? 'not defined');
1.14.1. Short if
function a() { return [ 'a' => '123' ]; } // standalone with one condition. No parentheses needed if ( $b = a() ) { print_r( $b, 1 ); } // use parentheses for multiple conditions if ( ( $b = a() ) && ( '321' !== $c = a() ) ) {} // a useful example if ( is_callable( 'a' ) && ( $b = a() ) && isset( $b['a'] ) && $b['a'] == '123' ) {} // $b is available var_dump($b); // Combine with ternory function abc($a) { return ($a) ? "Hello {$a}" : ''; } $a = ($a = abc('')) ? $a : 'No no'; // No no $a = ($a = abc('abc')) ? $a : 'No no'; // Hello abc
1.15. Short Echo
- 1.2.3.9 is not required since PHP 5.4.0
<?php $a = 'abc'; function abc($text) { return "running a function that returns $text"; } ?> <?= $a ?> <!-- eq. to --> <?=$a?> <!-- eq. to --> <?= $a; ?> <!-- eq. to --> <?php echo $a; ?> <!-- running a funciton that returns hello! --> <?= abc('hello!') ?> <!-- abc_123 --> <?= "{$a}_123" ?> <!-- 'false' --> <?= (0===1) ? 'true' : 'false' ?> <!-- 'true' --> <?= ($a === 'abc') ? 'true' : 'false' ?> <?= ($a === 'abc') ? <<<HTML <span> true </span> HTML : 'false' ?>
1.16. Error Control Operator
- All PHP versions
- https://www.php.net/manual/en/language.operators.errorcontrol.php
- prepend it to variables, function and
includecalls- Don't append to function/class definition and conditional structures e.g. if, foreach
- It temporarily sets the error reporting level to 0 for the code
- If an error is triggered, the error handler is still called but just with error level of 0
- Set and reset INI setting back and forth
- Drawbacks
- It's slow
- Refer to php:ini:error_reporting
$user = []; $a = @$user['id']; // by default, no error is output and $a is NULL $a = (int) @$user['id']; // int 0
1.17. Spaceship Operator
- Since PHP 7
- returns -1 ($a less than $b), 0 or 1
1.18. Multibyte php:multibyte
- All functions
- https://www.php.net/manual/en/ref.mbstring.php
- (no term)
- Functions
mb_*e.g.mb_strlen( 'abc' )is mb version ofstrlen - (no term)
str_replace()works with multibyte already- (no term)
- Functions work for mb only
- php:mb_strimwidth
php:ellipsis
mb_strimwidth ( string $str , int $start , int $width [, string $trimmarker = "" [, string $encoding = mb_internal_encoding() ]] ) : stringecho mb_strimwidth(get_the_title(), 0, 50, '...');
1.18.1. Remove 4 byte characters php:4byte
- PHP UTF-8 character range is wider than MySQL utf8. MySQL's true UTF-8 is utf8mb4 but sometimes it's hard to change
Here's how to remove any 4 byte character:
function replace4byte( $string ) { return preg_replace( '%(?: \xF0[\x90-\xBF][\x80-\xBF]{2} # planes 1-3 | [\xF1-\xF3][\x80-\xBF]{3} # planes 4-15 | \xF4[\x80-\x8F][\x80-\xBF]{2} # plane 16 )%xs', '', $string ); } var_dump( replace4byte( 'd' ), replace4byte( 'd_Emoji_d' ) );
1.19. Array
1.19.1. Spread Operator / Splat
// Spread Operator in Array Expression (Splat Operator) since PHP 7.4 $arr1 = [1, 2, 3]; $arr2 = [ 'a' => 123, 'b' => array( 'c' => 123 ), ]; var_dump([...$arr1] === [1, 2, 3]); var_dump(array(...$arr1) === [1, 2, 3]); var_dump([0, ...$arr1] === [0, 1, 2, 3]); var_dump(array(0, ...$arr1) === [0, 1, 2, 3]); var_dump([...$arr1, 0] === [1, 2, 3, 0]); // Fatal error: Unpacking arrays with string keys is forbidden for PHP < 8.1 // var_dump([...$arr2, 0] === [1, 2, 3, 0]); // Maybe eq. to the following for PHP < 8.1 var_dump(array_merge($arr2, array('c' => 321)) === ['a' => 123, 'b' => ['c' => 123], 'c' => 321]); // Argument Unpacking and Variadic Functions since 5.6 // - Applies to array or Traversable $args1 = [1, 2, 3]; $args2 = [4, 5, 6]; // $args is called a variadic parameter function test(...$args) { return $args; } var_dump(test(1, 2, 3) === $args1); var_dump(test(...[1, 2, 3]) === $args1); var_dump(test(...$args1) === $args1); var_dump(test(...new ArrayIterator($args1)) === $args1); var_dump(test(...$args1, ...$args2) === array_merge($args1, $args2)); var_dump(test(1, 2, 3, ...$args2) === array_merge($args1, $args2)); var_dump(test(...array_map('htmlspecialchars', $args))); // Not possible to use normal arguments after argument unpacking was used. These are invalid // test(...$args1, 4, 5, 6); // test(...$args1, 4, 5, 6, ...$args2);
1.19.2. list, compact, extract
$abNumeric = array('abc', 'xyz'); $abAssociative = array( 'a' => 'abc', 'b' => 'xyz', ); var_dump((list($a, $b) = $abNumeric) === $abNumeric); var_dump($a === 'abc'); list($a, $b) = $abAssociative; var_dump($a === null); list($a, $b) = array_values($abAssociative); var_dump($a === 'abc'); // No warning or notice is thrown. Good! list($a, $b) = null; var_dump($a === null, $b === null); // PHP 7.1.0+ // Order of items in list for associative array does not matter, but don't repeat the key $listReturn = list('a' => $a, 'b' => $b) = $abAssociative; var_dump($a === 'abc'); var_dump($listReturn === $abAssociative); $a = ['a' => 'abc', 'c' => 123]; var_dump(list('c' => $r1, 'b' => $r2, 'a' => $r3) = $a); // Notice is thrown as $a['c'] cannot be found var_dump($r1 === 123, $r2 === null, $r3 === 'abc'); list('a' => $a, 'b' => $b, 'e' => $e) = $abcde; var_dump(compact('a', 'b', 'e')); $a = 'abc'; $b = 'xyz'; var_dump(compact('a', 'b') === $abAssociative); // extract(array &$array, int $flags = EXTR_OVERWRITE, string $prefix = ""): int // Returns the number of variables successfully imported into the symbol table. $size = "large"; $var_array = array( "color" => "blue", "size" => "medium", "shape" => "sphere", ); extract($var_array, EXTR_PREFIX_SAME, "wddx"); var_dump($color === 'blue', $size === 'large', $shape === 'sphere', $wddx_size === 'medium');
1.19.3. array_merge, array_merge_recursive php:array_merge
array_merge$a = array( 321 => array('a' => 1, 'b' => 2), ); $b = array( 123 => array('a' => 3, 'b' => 4), ); $c = array( 123 => array('a' => 5, 'b' => 6), 987 => array('a' => 7, 'b' => 8), ); var_dump( $test = array_merge($a, $b, $c), $test == array( 0 => array('a' => 1, 'b' => 2), 1 => array('a' => 3, 'b' => 4), 2 => array('a' => 5, 'b' => 6), 3 => array('a' => 7, 'b' => 8), ) ); // Use array_replace or + for numeric array var_dump( $test = array_replace($a, $b, $c), $test === array( 321 => array('a' => 1, 'b' => 2), 123 => array('a' => 5, 'b' => 6), 987 => array('a' => 7, 'b' => 8), ) ); var_dump( $test = $a + $b + $c, $test === array( 321 => array('a' => 1, 'b' => 2), 123 => array('a' => 3, 'b' => 4), 987 => array('a' => 7, 'b' => 8), ) ); $a = array( 'a' => array('a' => 1, 'b' => 2), ); $b = array( 'b' => array('a' => 3, 'b' => 4), ); $c = array( 'b' => array('a' => 5, 'b' => 6), 'c' => array('a' => 7, 'b' => 8), ); var_dump( $test = array_merge($a, $b, $c), $test == array( 'a' => array('a' => 1, 'b' => 2), 'b' => array('a' => 5, 'b' => 6), 'c' => array('a' => 7, 'b' => 8), ) ); // array_replace is equivalent to array_merge for string-keyed array var_dump( $test = array_replace($a, $b, $c), $test === array( 321 => array('a' => 1, 'b' => 2), 123 => array('a' => 5, 'b' => 6), 987 => array('a' => 7, 'b' => 8), ) ); var_dump( $test = $a + $b + $c, $test == array( 'a' => array('a' => 1, 'b' => 2), 'b' => array('a' => 3, 'b' => 4), 'c' => array('a' => 7, 'b' => 8), ) );
- only merge on the first level
- If key is the same, value at later array will overwrite the first array's value
- If key does not exist in earlier arrays, the key/value will be appended
However, if key is numeric, values will always be appended (not overwriting) disregarding the values are the same
- Use this to dedup for numeric array merging
array_unique(array_merge($array1,$array2), SORT_REGULAR)
- refer to 1.19.30
array_merge_recursive- If key is the same, values will be appended as array
- Very different from array_merge
1.19.4. Append an array or an element to another array
1.19.4.1. Append an element
$a = array('1', 123); var_dump(array_push($a, 'hello') === 3); var_dump(($a[] = 'world') === 'world'); var_dump( array_merge($a, array('array_merge')) === array('1', 123, 'hello', 'world', 'array_merge') );
1.19.4.2. Append several elements array_push($a, 'hello', 'world')
1.19.4.3. Append an array
- If 2 arrays only have numeric keys e.g.
['a', 1, 2], php:array_merge can be used to append Use this for appending 2 arrays
$a = $b = [ ['a'=>1,'c'=>321], ['a'=>1, 'b'=>2] ]; $php56Less = $a; foreach ($b as $item) { $php56Less[] = $item; } var_dump($php56Less, '$php56Less'); $php56AndPlus = $a; array_push($php56AndPlus, ...$b); var_dump($php56AndPlus, '$php56AndPlus'); // splat operator since PHP 5.6 $array3 = [ 'prepend', ...$array1, 'inTheMiddle', ...$array2, 'append', ];
1.19.5. Union $array1 + $array2
$a = [ 'one' => 'a1', 'two' => 'a2', 'three' => 'a3' ]; $b = [ 'two' => 'b2', 'three' => 'b3', 'four' => 'b4' ]; var_dump($a + $b); $r = [ 'one' => 'a1', 'two' => 'a2', 'three' => 'a3', 'four' => 'b4' ];
1.19.6. Last Element, First Element, First Key
// First element/value // PHP 5.4+ $first = array_values($array)[0]; // throw error if array is empty // PHP 7.3+ $first = $array[array_key_first($array)]; // array_key_first($param) requires $param to be array otherwise warning will be thrown // array_key_first([]) returns null // Use this $first = is_array($array) ? ($array[array_key_first($array)] ?? "default value if $array is an empty array") : "default value if $array is not an array"; // $first = current($array); // requires the array current pointer is at the beginning // $first = reset($array); // mutates the array (set the current pointer to the first element) and returns the first array element value or false if the array is empty! // Last element $last = end($target_array); // set array pointer to last $last = reset($target_array); // PHP 7.3+ (is_array($array) && $array) ? $array[array_key_last($array)] : 'variable is empty or invalid';
1.19.7. array_shift, array_unshift prepend elements, array_pop
// array_pop ( array &$array ) : mixed // pop and return the value of the last element of array, shorten the array by one element // array_shift ( array &$array ) : mixed // first element, literal keys won't be affected // array_unshift ( array &$array [, mixed $... ] ) : int // prepend passed elements to the front of the array. Literal keys won't be changed $queue = ['orange', 'banana']; $queue2 = $queue; array_unshift($queue, 'apple', 'raspberry'); // ['apple', 'raspberry', 'orange', 'banana'] // to prepend into associative array, use array union (+) or array_merge $toAdd = ['apple']; // no mutation using array_merge for numeric arrays $queue2 = array_merge($toAdd, $queue2); // ['apple', 'orange', 'banana']
1.19.8. in_array
in_array ( mixed $needle , array $haystack [, bool $strict = FALSE ] ) : bool
- if needle is a string and the array is
string[], then case-sensitive comparison - Loose comparison unless $strict is set
- also check the types of the needle in the haystack
var_dump(in_array(null, null) === null); var_dump(in_array(3, null) === null); var_dump(in_array(null, [0, 1]) === true); var_dump(in_array('', [0, 1]) === true); var_dump(in_array('1', [0, 1]) === true); var_dump(in_array('3', [0, 1]) === false); var_dump(in_array('1', [0, 1], true) === false); // best solution var_dump(in_array(intval('3'), [0, 1]) === false);
1.19.9. Search by a value and return keys
Case insensitive search
$a= array( 'k1'=> 'one', 'k2' => 'one1', 'k3' => 'three', ); print_r( preg_grep( "/ONe$/i" , $a ) ); // return all matches // Array ( [k1] => one, ... ) // empty array if no match
- Use in_array for case sensitive search
1.19.10. Search by keys and return values in order
function arrayFilterByKeysInOrder($array, $keys, $default = null, $valuesOnly = true, $notToSetIfNotExists = false) { $allKeys = array_keys($array); $array2 = []; if ($notToSetIfNotExists) { return array_intersect_key($array, array_flip($keys)); } foreach ($keys as $k) { $r = (in_array($k, $allKeys)) ? $array[$k] : $default; if ($valuesOnly) { $array2[] = $r; } else { $array2[$k] = $r; } } return $array2; } $a = [ 'a' => 123, 'b' => 'abc', 'c' => 321, ]; var_dump( arrayFilterByKeysInOrder($a, ['c', 'abc', 'a']) == [ 321, null, 123 ] ); var_dump( arrayFilterByKeysInOrder($a, ['c', 'abc', 'a'], null, false) == [ 'c' => 321, 'abc' => null, 'a' => 123 ] ); var_dump( arrayFilterByKeysInOrder($a, ['c', 'abc', 'a'], null, false, true) == [ 'c' => 321, 'a' => 123, ] );
1.19.11. Search by value and return the first key by value array_search
array_search ( mixed $needle , array $haystack [, bool $strict = FALSE ] ) : mixed
- return
- the first key that has the value or false
- (no term)
- Case-insensitve
- (no term)
- Examples
Search value in a multidimensional array
$db = array( array('uid' => '100'), array('uid' => 100, 'optional' => 123), ); // PHP 5.5+ to support array_column var_dump(array_search('100', array_column($db, 'uid')) === 0); var_dump(array_search(100, array_column($db, 'uid')) === 0); // every item in $db has to have the column/array key as uid, otherwise, array_search returns the index of the first row that does not have the column/array key var_dump(array_search(123, array_column($db, 'optional')) === 0); // PHP < 5.5 var_dump(array_search('100', array_map(function ($v) { return $v['uid']; }, $db)) === 0); var_dump(array_search(100, array_map(function ($v) { return $v['uid']; }, $db)) === 0); var_dump(array_search(123, array_map(function ($v) { return $v['optional']; }, $db)) === 1);
$key = array_search('100', array_column($userdb, 'uid'));Since PHP 5.5- every item in $userdb has to have the column/array key as uid, otherwise, array_search returns the index of the first row that does not have the column/array key
1.19.12. Delete by a value
- Case-sensitive
foreach (array_keys($items, 'search this value', true) as $key) { // true is case-insensitive unset($items[$key]); } // Comparison is done by `(string) $elem1 === (string) $elem2` $items = array_diff($items, ['search this value']); // PHP 7.4 $items = array_filter($items, fn ($m) => $m != 'search this value');
1.19.13. Change Case for All Keys
$_lower_GET = array_change_key_case($_GET, CASE_LOWER); // Or CASE_UPPER
1.19.14. Change keys only
- The length of
$keysand$valuesshould be equal
array_combine(array $keys, array $values): array
$numeric_array = array(1, 2, 3); $assign_keys = array('a', 'b', 'c'); array_combine($assign_keys, $numeric_array) === array ( 'a' => 1, 'b' => 2, 'c' => 3, ); $a = array( 'a_321' => '123', 'abc' => '321', ); // Uppercase the first letter in keys var_export($test = array_combine(array_map('ucfirst', array_keys($a)), $a)); var_dump( $test === array( 'A_321' => '123', 'Abc' => '321', ) ); // Remove _ in keys var_export( $test = array_combine( array_map( function ($item) { return str_replace('_', '', $item); }, array_keys($a) ), $a ) ); var_dump( $test === array( 'a321' => '123', 'abc' => '321', ) ); $indexDBRows = array( 0 => array('id' => 123, 'name' => 'Li'), 1 => array('id' => 124, 'name' => 'Hello'), 2 => array('id' => 125, 'name' => 'World'), ); $primaryKeyDBRows = array( 123 => array( 'id' => 123, 'name' => 'Li', ), 124 => array( 'id' => 124, 'name' => 'Hello', ), 125 => array( 'id' => 125, 'name' => 'World', ), ); // PHP 5.5+ $r = array_combine(array_column($indexDBRows, 'id'), $indexDBRows); var_dump($r === $primaryKeyDBRows); // PHP < 5.5 $r = array_combine( array_map(function($v) { return $v['id']; }, $indexDBRows), $indexDBRows ); var_dump($r === $primaryKeyDBRows);
1.19.14.1. Convert Numeric Arrays to Associative Arrays
function arrayNumericToAssociative(array $keys, array $array, $options = array()): array { array_walk( $keys, function ($v, $k) use (&$r, $array, $options) { if (!isset($v)) { return $r; } $r[$v] = array_key_exists($k, $array) ? $array[$k] : ($options['default'] ?? null); return true; }, $r = array() ); return $r; } $keys = array('a', 'b', 'c'); arrayNumericToAssociative($keys, [1, 2, 3, 4]) === ['a' => 1, 'b' => 2, 'c' => 3]; arrayNumericToAssociative($keys, [1, 2]) === ['a' => 1, 'b' => 2, 'c' => null]; arrayNumericToAssociative($keys, []) === ['a' => null, 'b' => null, 'c' => null];
1.19.15. Get values of a column in a multidimensional array
Since PHP 5.5
$records = [ [ 'id' => 2135, 'first_name' => 'John', 'last_name' => 'Doe', ], [ 'id' => 3245, 'first_name' => 'Sally', 'last_name' => 'Smith', ], [ 'id' => 5342, 'first_name' => 'Jane', // Notice! there's no last_name ], [ 'id' => 5623, 'first_name' => 'Peter', 'last_name' => 'Doe', ], ]; $a1 = array_column($records, 'last_name'); // ['Doe', 'Smith', 'Doe'] $a2 = array_column($records, 'last_name', 'id'); // [2135 => 'Doe', 3245 => 'Smith', 5623 => 'Doe'] array_column([], 'id') === []; array_column(null, 'id') === null; // no __get and __isset() magic methods class User { public $username; public function __construct(string $username) { $this->username = $username; } } $users = [ new User('user 1'), new User('user 2'), new User('user 3'), ]; $a3 = array_column($users, 'username'); // ['user 1', 'user 2', 'user 3'] // __get and __isset() magic methods and get private properties class Person { private $name; public function __construct(string $name) { $this->name = $name; } public function __get($prop) { return $this->$prop; } public function __isset($prop): bool { return isset($this->$prop); } } $people = [ new Person('Fred'), new Person('Jane'), new Person('John'), ]; $a4 = array_column($people, 'name'); // ['Fred', 'Jane', 'John']
1.19.16. array_diff, array_diff_key, array_diff_assoc, array_intersect, array_intersect_assoc
- Use
array_diff,array_diff_key,array_intersectfor numeric index array - Use
array_diff_assoc,array_intersect_assocfor associative array
array_diff(array $array, array ...$arrays): array
- Compare on values, and return the key/value pairs in the first array for the values don't exist in the following array(s)
- Two elements are considered equal if and only if
(string) $elem1 === (string) $elem2 - Throw Notice when element cannot be cast to a string
- Can be used to remove elements of
$arraySourcethat are in$arrayValuesToRemove- To keep original index
array_diff($arraySource = [312, 401, 15, 401, 3], $arrayValuesToRemove = [401]) === [0 => 312, 2 => 15, 4 => 3]- Continuous index
array_values(array_diff([312, 401, 15, 401, 3], [401])) === [312, 15, 3]
array_intersect(array $array, array ...$arrays): array array_intersect_assoc(array $array, array ...$arrays): array
- Return an array containing all the values of the first array that are present in all other arrays
array_intersect_key(array $array, array ...$arrays): array
- Computes the intersection of arrays using keys for comparison
Filter an array by keys
$my_array = array("foo" => 1, "hello" => "world"); $allowed = array("foo", "bar"); array_intersect_key($my_array, array_flip($allowed)) === array("foo" => 1);
Single dimension
$arr = ['nice_item', 'remove_me', 'another_liked_item', 'remove_me', 'remove_me_also']; $arr = array_diff($arr, ['remove_me', 'remove_me_also']); // [ 0 => 'nice_item', 2 => 'another_liked_item'] $hasKeysThatAreNotSupposedToHave = ['nice_item', 'new key not existing']; $diff = array_diff($hasKeysThatAreNotSupposedToHave, $arr); // [ 1 => 'new key not existing' ] // Check if $arr has all these keys $hasAllKeys = ['nice_item', 'another_liked_item', 'another_liked_item']; // keys can be repeated $missingKeys = array_diff($hasAllKeys, $arr); if (empty($missingKeys)) { echo 'Has all keys'; } else { var_dump($missingKeys, 'These keys are missing'); }
Associative arrays
$array1 = array('blue' => 1, 'red' => 2, 'green' => 3, 'purple' => 4); $array2 = array('green' => 5, 'yellow' => 7, 'cyan' => 8); $array3 = array('green' => 3, 'yellow' => 7, 'cyan' => 8); var_export( array_diff($array1, $array2) === array( 'blue' => 1, 'red' => 2, 'green' => 3, 'purple' => 4, ) ); var_export( array_diff($array1, $array3) === array( 'blue' => 1, 'red' => 2, 'purple' => 4, ) ); var_export( array_diff_key($array1, $array2) === array( 'blue' => 1, 'red' => 2, 'purple' => 4, ) );
1.19.16.1. array_diff_key - Compare on keys
array_diff_key(array $array, array ...$arrays): array
$values = [ 'a' => 123, 'b' => 321, 'c' => 'abc', ]; $keysWithDummyValues = array_fill_keys(['a', 'z', 'c'], 0); var_dump($keysWithDummyValues === ['a' => 0, 'c' => 0, 'z' => 0]); var_dump(array_diff_key($values, $keysWithDummyValues) === ['b' => 321]); var_dump(array_diff_key($values, []) === $values); var_dump(array_diff_key($values, null) === false); $keysWithDummyValues2 = array_fill_keys(['x', 'y', 'b'], 0); var_dump(array_diff_key($values, $keysWithDummyValues2) === ['a' => 123, 'c' => 'abc']);
1.19.17. array_key_exists
- See 1.19.16
function array_key_exists($key, array $search) bool $a = array( 'a' => 123, 'b'=> 321, ); // Check if a single key exists in an array var_export(array_key_exists('a', $a) === true); // Check if multiple keys all exist in an array function arrayKeysAllExist($keys, $arr, $onlyThoseKeys = false): bool { if ($onlyThoseKeys) { $arrKeys = array_keys($arr); $diff1 = array_diff($keys, $arrKeys); return $diff1 == array() && array_diff($arrKeys, $keys) == $diff1; } return !array_diff_key(array_flip($keys), $arr); } var_dump(arrayKeysAllExist(array('a', 'c'), $a) === false); var_dump(arrayKeysAllExist(array('a'), $a) === true); var_dump(arrayKeysAllExist(array('a', 'a'), $a) === true); var_dump(arrayKeysAllExist(array('a', 'a'), $a, true) === false); function array_keys_exist_count(array $keys, array $arr) { return count(array_intersect_key(array_flip($keys), $arr)); } var_export(array_keys_exist_count(array('a', 'c'), $a) == 1);
1.19.18. array_map
- Do something with each array element's value and return a new array
- See array_combine to change keys
array_map(?callable $callback, array $array, array ...$arrays): array
function sanitize($s) { return htmlspecialchars($s); } $a = array('harmless', '<bad>', '>>click here!<<'); $a = array_map('sanitize', $a); // immutable // trim each element $a = array_map('trim', $a); echo implode(' ', $a); // Lambda function is available since PHP 5.3.0 $func = function($value) { return $value * 2; }; print_r(array_map($func, range(1, 5))); // multiple arrays can be passed $letters = [ 'a' => 'a-abc', 'b' => 'b-abc', ]; $numericArray = array_map( function ($letter, $index) { return "Index at $index is $letter"; }, // fn($letter, $index) => "Index at $index is $letter", // arrow function since PHP 7.4 $letters, array_keys($letters) ); var_dump($numericArray === ['Index at a is a-abc', 'Index at b is b-abc']);
1.19.19. array_splice Insert to any position of an array
1.19.19.1. Example
$original = array('a', 'b', 'c', 'd', 'e'); $inserted = array('x', 'y'); // remove and replace elements in array // array_splice (array &$input, $offset, $length = null, $replacement = null) // return the array consisting of the extracted elements. $extractedElements = array_splice($original, 3, 0, $inserted); var_dump( 'Length is 0 means to insert at position 3 (offset)', $original === array('a', 'b', 'c', 'x', 'y', 'd', 'e') ); var_dump( 'Extracted elements is empty array because only insert', $extractedElements === array() ); array_unshift($original, 'f', 'g'); var_dump( 'Prepend f and g in front', $original === array('f', 'g', 'a', 'b', 'c', 'x', 'y', 'd', 'e') ); $insertedOne = 'a string'; $extractedElements = array_splice($original, 3, 0, $insertedOne); var_dump( 'Inserted One', $original === array('f', 'g', 'a', 'a string', 'b', 'c', 'x', 'y', 'd', 'e') ); $original = array('a' => 1, 'b' => 2, 'c' => 3, 'd' => 4, 'e' => 5); $inserted = array('x' => 6, 'y' => 7); $extractedElements = array_splice($original, 3, 0, $inserted); var_dump( 'Associative array', $original === array( 'a' => 1, 'b' => 2, 'c' => 3, 0 => 6, 1 => 7, 'd' => 4, 'e' => 5, ) ); var_dump( 'array_keys($original)', array_keys($original) === ['a', 'b', 'c', 0, 1, 'd', 'e'], 'Find Index By Key', findIndexByKey($original, 'c') === 2, findIndexByKey($original, 'd') === 5, findIndexByKey($original, 1) === 4, findIndexByKey($original, '1') === false, findIndexByKey($original, 0) === 3, findIndexByKey($original, '0') === false, findIndexByKey(array_merge($original, array('d' => 6)), 'd') === 5, findIndexByKey(array('a' => 1, 'b' => 2, 'c' => 3, 'd' => 4, 'e' => 5, 'd' => 6), 'd') === 3 ); var_dump( 'Insert before key d', insertKeyValuePairsToAssocArray($original, 'd', $inserted) === array( 'a' => 1, 'b' => 2, 'c' => 3, 0 => 6, 1 => 7, 'x' => 6, 'y' => 7, 'd' => 4, 'e' => 5, ), 'Insert after key d', insertKeyValuePairsToAssocArray($original, 'd', $inserted, false) === array( 'a' => 1, 'b' => 2, 'c' => 3, 0 => 6, 1 => 7, 'd' => 4, 'x' => 6, 'y' => 7, 'e' => 5, ), 'Insert after key e (the last key in original array)', insertKeyValuePairsToAssocArray($original, 'e', $inserted, false) === array( 'a' => 1, 'b' => 2, 'c' => 3, 0 => 6, 1 => 7, 'd' => 4, 'e' => 5, 'x' => 6, 'y' => 7, ), 'Insert after key 1', insertKeyValuePairsToAssocArray($original, 1, $inserted, false) === array( 'a' => 1, 'b' => 2, 'c' => 3, 0 => 6, 1 => 7, 'x' => 6, 'y' => 7, 'd' => 4, 'e' => 5, ) ); function findIndexByKey($array, $key) { return array_search($key, array_keys($array), true); } var_export( insertKeyValuePairsToAssocArray( [1, 2, 3, 'x' => 'abc', 'y' => 1, 'z' => null], 'z', ['xyz' => true], false ) === [1, 2, 3, 'x' => 'abc', 'y' => 1, 'z' => null, 'xyz' => true] ); var_export( insertKeyValuePairsToAssocArray( [1, 2, 3, 'x' => 'abc', 'y' => 1, 'z' => null], 'z', ['xyz' => true, 'x' => 'override'], false ) === [1, 2, 3, 'x' => 'override', 'y' => 1, 'z' => null, 'xyz' => true] ); /** * Insert key/value pairs before or after a key/value * * @param array $array array_merge is used for numeric keys, please use it carefully. If array is purely string-keyed, no problem * @param string|int $insertAtThisKey * @param array $insertItems * @param bool $insertBefore otherwise, insert after * * @return array * @throws Exception */ function insertKeyValuePairsToAssocArray($array, $insertAtThisKey, $insertItems, $insertBefore = true): array { if (!array_key_exists($insertAtThisKey, $array)) { throw new Exception('Key cannot be found.'); } $keyAtIndex = findIndexByKey($array, $insertAtThisKey); $indexToInsert = $insertBefore ? $keyAtIndex : $keyAtIndex + 1; $arrayEnd = array_splice($array, $indexToInsert); $arrayStart = array_splice($array, 0, $indexToInsert); return array_merge($arrayStart, $insertItems, $arrayEnd); }
1.19.19.2. Example not using array_splice
$original = array( array( 'paymentid' => '2255997', 'subItems' => array( array( 'paymentid' => '2255995', ), array( 'paymentid' => '2255996', ), ), ), array( 'paymentid' => '2256997', 'subItems' => array( array( 'paymentid' => '2256995', ), array( 'paymentid' => '2256996', ), ), ), ); $result = array( array( 'paymentid' => '2255997', 'subItems' => array( array( 'paymentid' => '2255995', ), array( 'paymentid' => '2255996', ), ), ), array( 'paymentid' => '2255995', ), array( 'paymentid' => '2255996', ), array( 'paymentid' => '2256997', 'subItems' => array( array( 'paymentid' => '2256995', ), array( 'paymentid' => '2256996', ), ), ), array( 'paymentid' => '2256995', ), array( 'paymentid' => '2256996', ), ); $copy = []; foreach ($original as $key => $value) { $copy[] = $value; if (!($value['subItems'] ?? null)) { continue; } $itemsToInsert = $value['subItems']; // $copy[$key]['subItems'] = []; $copy = array_merge($copy, $itemsToInsert); } var_dump($copy === $result); $original = array( 2255997 => array( 'paymentid' => '2255997', 'subItems' => array( 2255995 => array( 'paymentid' => '2255995', ), 2255996 => array( 'paymentid' => '2255996', ), ), ), 2256997 => array( 'paymentid' => '2256997', 'subItems' => array( 2256995 => array( 'paymentid' => '2256995', ), 2256996 => array( 'paymentid' => '2256996', ), ), ), ); $result = array( 2255997 => array( 'paymentid' => '2255997', 'subItems' => array( 2255995 => array( 'paymentid' => '2255995', ), 2255996 => array( 'paymentid' => '2255996', ), ), ), 2255995 => array( 'paymentid' => '2255995', ), 2255996 => array( 'paymentid' => '2255996', ), 2256997 => array( 'paymentid' => '2256997', 'subItems' => array( 2256995 => array( 'paymentid' => '2256995', ), 2256996 => array( 'paymentid' => '2256996', ), ), ), 2256995 => array( 'paymentid' => '2256995', ), 2256996 => array( 'paymentid' => '2256996', ), ); $copy = []; foreach ($original as $key => $value) { $copy[$key] = $value; if (!($value['subItems'] ?? null)) { continue; } foreach ($value['subItems'] as $subKey => $subValue) { $copy[$subKey] = $subValue; } } var_dump($copy === $result);
1.19.20. array_slice Extract a slice of the array
array_slice(
array $array,
int $offset,
?int $length = null,
bool $preserve_keys = false
): array
$input = array("a", "b", "c", "d", "e"); $output = array_slice($input, 2); // returns "c", "d", and "e" $output = array_slice($input, -2, 1); // returns "d" $output = array_slice($input, 0, 3); // returns "a", "b", and "c"
1.19.21. array_fill fill values
- See range()
// array_fill ($start_index, $num, $value) var_dump( 'Fill from index 0 to 2', array_fill(0, 2, 'x') === array('x', 'x') ); var_dump( 'Fill from index 1 to 2', array_fill(1, 2, 'x') === array(1 => 'x', 2 => 'x') );
1.19.22. array_fill_keys
- See array_combine
$keys = array('foo', 5, 10, 'bar'); array_fill_keys($keys, 'banana') === array( 'foo' => 'banana', 5 => 'banana', 10 => 'banana', 'bar' => 'banana', );
1.19.23. array_walk
- Side-effect
- may mutate source array values. And only array values can be changed.
- Cannot change array keys! Use array_combine
- Cannot add, unset or reorder elements/keys in array
- Cannot change array keys! Use array_combine
- (no term)
- Can access source array key and value
- array_reduce can only access value, but it returns a value after the loop
- (no term)
- Return bool
- To return the new array, see array_map
- May use
returnin callback for guard clauses
- (no term)
- Will walk through the entire array regardless of pointer position
- (no term)
- No
breaknorcontinue
array_walk ( array|object &$array , callable $callback , mixed $userdata = null ) : bool $fruits = array("d" => "lemon", "a" => "orange", "b" => "banana", "c" => "apple"); array_walk($fruits, function($v, $k) { printf('%s="%s" ', $k, $v); }); // To alter $fruits array_walk($fruits, function(&$v, $k, $prefix) { $v = "$prefix: $v"; }, 'customPrefix'); var_dump('$fruits', $fruits); // Don't do this to alter external param $fruits2 !!! $fruits2 = []; array_walk($fruits, function($v, $k, &$externalVar) { $externalVar[$k] = $v; }, $fruits2); var_dump('$fruits2', $fruits2 === []); // To alter external params #1 array_walk($fruits, function($v, $k) use (&$fruits2) { $fruits2[$k] = $v; }, $fruits2 = []); var_dump('$fruits2', $fruits2); // works! // To alter external params #2 $fruits3 = []; array_walk($fruits, function($v, $k) use (&$fruits3) { $fruits3[$k] = $v; }); var_dump('$fruits3', $fruits3); // works! // Other external params can be passed (even as a reference) $counter = 0; array_walk($fruits, function($v, $k) use (&$fruits2, &$counter) { $fruits2[$k] = $v; $counter++; }, $fruits2 = []); var_dump('$fruits2', $fruits2, '$counter', $counter); // works! // Walk an existing array and create a new array $index array_walk($array, function ($v, $k) use (&$index) { $index[] = $k; }, $index = []); // Unset a subkey in a multi-dimensional array // unset subkey recognizedFields in a deep nested array array_walk($r['results'][$entityTitle]['items'], function ($v, $k) use (&$item) { unset($item[$k]['recognizedFields']); }, $r['results'][$entityTitle]['items']);
1.19.24. array_filter
Basics
array_filter(array $array, ?callable $callback = null, int $mode = 0): array
- cb function should return true or false. By default, only value is passed to the callback function. Unless flag is used
array_filter($my_array)can be used to roughly filter empty/null elements outarray element value == false$simpleXMLElementWithoutChildenNorAttributes = new SimpleXMLElement("<news></news>"); $a = array( '0', // empty '', // empty null, // empty 0.0, // empty -0.0, // empty 0, // empty false, // empty array(), // empty $simpleXMLElementWithoutChildenNorAttributes, // empty ' ', -1, ); var_export(array_diff($a, array_filter($a)) === array( 0 => '0', 1 => '', 2 => NULL, 4 => 0.0, 5 => -0.0, 6 => 0, 8 => false, 9 => array(), 10 => $simpleXMLElementWithoutChildenNorAttributes, ));
- Original index number is used!
- See 1.19.16 to filter an array by keys
$a = [ '1' => null, '2' => 0, '3' => '0', '4' => '', '5' => false, '6' => 123, '7' => -1 ]; var_dump(array_filter($a) === array('6' => 123, '7' => -1)); array_filter(explode(',', $string)); array_filter(explode(',', $string), 'strlen'); $arr = ['a' => 1, 'b' => 2, 'c' => 3, 'd' => 4]; // default values are passed. To pass the keys of array, use a flag var_dump(array_filter($arr, function($k) { return $k == 'b'; }, ARRAY_FILTER_USE_KEY)); var_dump(array_filter($arr, function($v, $k) { return $k == 'b' || $v == 4; }, ARRAY_FILTER_USE_BOTH)); $clients = [ 'Li' => 'hello', ]; $htmlOptions = ''; $fn = function ($d = '') { return $d; }; array_filter($clients, function ($v, $k) use (&$htmlOptions, $fn) { if (!$k) { return false; } $htmlOptions .= <<<HTML <option value="{$fn(htmlspecialchars($k))}">{$fn(htmlspecialchars($v))}</option> HTML; return false; }, ARRAY_FILTER_USE_BOTH); var_dump($htmlOptions);
1.19.25. array_reduce
See js:array:reduce
array_reduce ( array $array , callable $callback [, mixed $initial = NULL ] ) : mixed callback ( mixed $carry , mixed $item ) : mixed
$sum = 100; $fees = array( '1' => array('amount' => 1), '2' => array('amount' => 2), ); $r = array_reduce($fees, function($carry, $item) { return $carry -= $item['amount']; }, $sum); var_dump($r === 97);
- Return
- null
- if
$arrayis not an array and warning is thrown - if
$arrayis an empty array and$initialis not set
- if
- null
- Can be associative array, but no way to access the key in callback function
- Use array_walk but it suggests the input array might be mutated
- Use array_filter
- Return a value after loop
- Could be null if
$arrayis not an array
- Could be null if
Return deep value in array
/** * @param array $array * @param array|string|int $keys * @param null $default * * @return mixed|null */ function arrayGetDeepValue($array, $keys, $default = null) { $keys = ( ! is_array($keys)) ? array($keys) : $keys; $r = array_reduce( $keys, function ($result, $index) { return (isset($result) && array_key_exists($index, $result)) ? $result[$index] : null; }, $array ); return isset($r) ? $r : $default; }
1.19.26. array_sum()
function arraySubtract($array) { $r = null; array_walk( $array, function ($v) use (&$r) { $r = isset($r) ? $r - $v : $v; }, $r ); return $r; } var_dump($test = array_sum(array(100, 10, 20, 30)), $test === 100 + 10 + 20 + 30); var_dump($test = arraySubtract(array(100, 10, 20, 30)), $test === 100 - 10 - 20 - 30); var_dump($test = arraySubtract(array('a' => 100, 'b' => 10, 'c' => 20, 'd' => 30)), $test === 100 - 10 - 20 - 30); var_dump($test = array_sum(array('a' => 100, 'b' => 10, 'c' => 20, 'd' => 30)), $test === 100 + 10 + 20 + 30);
1.19.27. Sort arrays
1.19.27.1. sort
- Sort by value, the sorted array has keys starting from 0 (key/value pair is not maintained)
- sort
- ascending order
- rsort
- descending order
- Mutation
- Flags
- SORT_REGULAR
- SORT_NUMERIC
- SORT_STRING
- SORT_LOCALE_STRING
- SORT_NATURAL
- case-insensitive for SORT_STRING and SORT_NATURAL
sort ( array &$array [, int $sort_flags = SORT_REGULAR ] ) : bool
$fruits = array('d' => 'lemon', 'a' => 'orange', 'b' => 'banana', 'c' => 'apple'); sort($fruits); var_dump($fruits === array('apple', 'banana', 'lemon', 'orange')); $fruits = array('lemon', 'orange', 'banana', 'apple'); sort($fruits); var_dump($fruits === array('apple', 'banana', 'lemon', 'orange'));
1.19.27.2. ksort, krsort
- Sort (associative) array by its key and maintain key to data correlations
- ksort
- ascending order
- krsort
- descending order
- Mutation
- Return true on success or false on failure
krsort(array &$array, int $flags = SORT_REGULAR): bool
1.19.27.3. asort, arsort
- Sort (associative) array by its value and maintain key to data correlations
- asort
- ascending order
- arsort
- descending order
- Similar to 1.19.27.2
- When target array is numeric array, the key is maintained
- Mutation
- Return true on success or false on failure
arsort(array &$array, int $flags = SORT_REGULAR): bool
$sortedKeyValuePair = array ( 'a' => 'orange', 'd' => 'lemon', 'b' => 'banana', 'c' => 'apple', ); $fruits = array('d' => 'lemon', 'a' => 'orange', 'b' => 'banana', 'c' => 'apple'); arsort($fruits); var_dump($fruits, $fruits === $sortedKeyValuePair);
1.19.27.4. usort, uasort - custom sort by value
- Mutation
- Sort by value
usortdoesn't maintain key association butuasortdoes. If callback function returns negative integer, means the comparison is less than- See Spaceship operator
- Basic example
$a = array( array('id' => 321, 'name' => 'Li Li 321'), ); $b = array( array('id' => 456, 'name' => 'Li Li 456'), array('id' => 123, 'name' => 'Li Li 123'), ); $c = array(); // no warning $d = array( array('name' => 'Li Li Random no id'), // PHP Notice: no index 'id' ); $toSort = array_merge($a, $b); $toSort2 = array_merge($b, $c); $toSort3 = array_merge($b, $d); function mySort($a, $b) { return $a['id'] - $b['id']; // ascending order } usort($toSort, 'mySort'); var_export($toSort); var_dump( $toSort === array( array('id' => 123, 'name' => 'Li Li 123',), array('id' => 321, 'name' => 'Li Li 321',), array('id' => 456, 'name' => 'Li Li 456',), ) ); usort($toSort2, 'mySort'); var_export($toSort2); var_dump( $toSort2 === array( array('id' => 123, 'name' => 'Li Li 123'), array('id' => 456, 'name' => 'Li Li 456'), ) ); usort($toSort3, 'mySort'); var_export($toSort3); var_dump( $toSort3 === array( array('name' => 'Li Li Random no id'), // PHP Notice: no index 'id' array('id' => 123, 'name' => 'Li Li 123'), array('id' => 456, 'name' => 'Li Li 456'), ) );
- More examples
$to_sort = [ [ 'hashtag' => 'a7e87329b5eab8578f4f1098a152d6f4', 'title' => 'Flower', 'order' => 3 ], ['hashtag' => 'b24ce0cd392a5b0b8dedc66c25213594', 'title' => 'Free', 'order' => 2 ], [ 'hashtag' => 'e7d31fc0602fb2ede144d18cdffd816b', 'title' => 'Ready', 'order' => 1 ] ]; // PHP 5.2 and lower function sortByOrder($a, $b) { return $a['order'] - $b['order']; // ascending order } usort($myArray, 'sortByOrder'); // Since PHP 5.3 usort($myArray, function($a, $b) { return $a['order'] - $b['order']; }); // Since PHP 7 usort($myArray, function($a, $b) { return $a['order'] <=> $b['order']; }); // Order by multiple values usort($myArray, function($a, $b) { $retval = $a['order'] <=> $b['order']; if ($retval == 0) { $retval = $a['suborder'] <=> $b['suborder']; if ($retval == 0) { $retval = $a['details']['subsuborder'] <=> $b['details']['subsuborder']; } } return $retval; });
1.19.27.5. uksort - custom sort by keys
1.19.28. array_reverse
- $preserve_keys
- if true, numeric keys are preserved. Non-numeric keys are not affected by this setting and will always be preserved
array_reverse ( array $array [, bool $preserve_keys = FALSE ] ) : array
1.19.29. array_keys, array_values
- array_keys also has filter by array value and strict type matching (
=) options
array_keys ( array $array , mixed $search_value [, bool $strict = FALSE ] ) : array array_values ( array $array ) : array
1.19.30. array_unique
array_unique ( array $array [, int $sort_flags = SORT_STRING ] ) : array
- return a new array without duplicate values
- sort_flags
- SORT_STRING
- default. the same if
(string) $elem1 === (string) $elem2 - SORT_REGULAR
- don't change types. Refer to php:array_merge
- SORT_NUMERIC
- compare items numerically
- SORT_LOCALE_STRING
- compare items as strings, based on the current locale
- Not intended to work on multi dimensional arrays
- Keys are preserved, the first element with the same value is retained
Example: reverse mapping
$map = [ 'Annually' => 12, 'Semi-Annually' => 6, 'Quarterly' => 6, // no mapping on the receiving end 'Monthly' => 1, 'Other' => 1 ]; // norway of mapping $fromPartyA = 'Quarterly'; $toUs = $map[ $fromPartyA ]; // reverse mapping $reversedMap = array_flip( array_unique( $map ) ); $fromUs = 6; $toPartyA = $reversedMap[ $fromUs ]; // Semi-Annually
Sometimes continuous keys is desired for numeric array, use
array_keys// array_flip make sure the returned keys are unique $unique=array_keys(array_flip(['a', 'a', 'b', 'c'])); // ['a', 'b', 'c']
1.19.31. array_flip
- Return an array with keys eq. to the values of input array, with values eq. to the keys of the input array
- return NULL on failure
- The new keys need to be valid e.g. integer or string. Invalid key and its value will not be included in the result
- The latest key will be used as its value
var_dump( array_flip( [ 'a', 'a', 'b', 'c' ] ) ); // [ a => 1, b => 2, c=> 3 ] var_dump( array_flip( [ 'a', 'b', 'a', 'c' ] ) ); // [ a => 2, b => 1, c => 3]
1.19.32. range
// When start and end are numbers var_export(range(2,2) === array(2)); var_export(range(2,3) === array(2, 3)); // String of numbers are converted to numbers in the result var_export(range('2','3') === array(2, 3)); // default step is int 1 var_export(range(2,3, 1) === array(2, 3)); // step is int 3 var_export(range(2,10, 3) === array(2, 5, 8)); // can go backwards, with step being positive int 1, only based on start and end var_export(range(2,1) === array(2, 1)); // step should be a positive number, can be float var_export(range(2,2.2, 0.1) === array(2.0, 2.1, 2.2)); // String of numbers are converted to numbers in the result var_export(range('2','2.2', 0.1) === array(2.0, 2.1, 2.2)); // Force the result to be string of numbers instead of numbers, although it's not perfect for floats... // It works for strings of integers var_export( array_map('strval', range('2', '2.2', 0.1)) === array( '2', // it should be '2.0' as the original.. but oh well.. '2.1', '2.2', ) ); // When start and end are single length characters var_export(range('a','c') === array('a', 'b', 'c')); var_export(range('a','c', 2) === array('a', 'c')); // The character length of start and end should one, the following does not work though does not throw errors var_export(range('AB','AC')); // Use case: Generate a series of strings with prefix and number padding var_export( array_map(function ($n) { return sprintf('sample_%03d', $n); }, range(1, 2)) === array('sample_001', 'sample_002') );
1.19.33. Associative array
function isAssoc(array $arr) { if (array() === $arr) return false; return array_keys($arr) !== range(0, count($arr) - 1); }
1.19.34. Set array value by deep path
$a = [ 'a' => [ 'b' => 123, ], ]; function setArrayByPath(&$array, $path, $value): array { $temp = &$array; foreach ($path as $key) { // if key does not exist, the value of that key is NULL $temp = &$temp[$key]; } $temp = $value; return $array; } var_dump(($arr = setArrayByPath($a, ['a', 'b'], 'some value')) === [ 'a' => [ 'b' => 'some value', ], ]); setArrayByPath($a, ['z', 'y', 'x'], 'some value'); var_dump($a === [ 'a' => [ 'b' => 'some value', ], 'z' => [ 'y' => [ 'x' => 'some value', ], ], ]);
1.19.35. If it's numeric and continuous array
function hasMany($arr) { return array_keys($arr) === range(0, count($arr) - 1); }
1.19.36. Custom pretty print array
function printData($array) { echo "<ul>"; foreach ($array as $key => $value) { echo "<li>$key"; if (is_array($value)) { printData($value); } else { echo ": " . htmlspecialchars($value); } echo "</li>"; } echo "</ul>"; } printData($data);
1.20. String
1.20.1. Conversion to string
1.20.2. Empty string
// All empty values $testCase = array( '', "", null, array(), FALSE, NULL, '0', 0, ); if (trim($str) != '') {}
1.20.3. Multibyte - refer to php:multibyte
1.20.4. String to HTML,
1.20.4.1. Parse
$dom = new DOMDocument; $dom->loadHTML($xml); $imgs = $dom->getElementsByTagName('img'); foreach ( $imgs as $img ) { echo $img->nodeValue, PHP_EOL; echo $img->getAttribute('src'); } // To get the first img $_first_img = $imgs->item(0); echo $_first_img->getAttribute('src');
1.20.4.2. Strip one specific HTML tag
$dom = new DOMDocument; $dom->loadHTML($htmlString); $xPath = new DOMXPath($dom); $nodes = $xPath->query('//*[@id="anotherDiv"]'); if($nodes->item(0)) { $nodes->item(0)->parentNode->removeChild($nodes->item(0)); } $list = $doc->getElementsByTagName("p"); while ($list->length > 0) { $p = $list->item(0); $p->parentNode->removeChild($p); } echo $dom->saveHTML();
1.20.4.3. Add element by adding fragment
The above will add <!doctype><html><body> tags, use this to do fragment
$dom = new DOMDocument; $fragment = $dom->createDocumentFragment(); $fragment->appendXML($form); $dom->appendChild($fragment); $noscripts = $dom->getElementsByTagName('noscript'); while ($noscripts->length > 0) { $i = $noscripts->item(0); $i->parentNode->removeChild($i); } return $dom->saveHTML();
1.20.4.4. Output
$doc = new DOMDocument(); // set before loading elements $doc->preserveWhiteSpace = false; $doc->formatOutput = true; $doc->loadHTML($r); $doc->saveHTMLFile($filePath);
1.20.5. Regex
1.20.5.1. See 8.6.9.2.1
1.20.5.3. preg_match
preg_match ( string $pattern , string $subject [, array &$matches [, int $flags = 0 [, int $offset = 0 ]]] ) : int
- Return 1 if the pattern matches given subject, 0 if it does not, or FALSE if an error occurred
$matches- $matches[0]
- the text matched the full pattern
- $matches[1]
- the text matched the first captured parenthesized subpattern and so on
1.20.5.4. preg_match_all
preg_match_all ( string $pattern , string $subject [, array &$matches [, int $flags = PREG_PATTERN_ORDER [, int $offset = 0 ]]] ) : int
- Return the number of full pattern matches (could be 0). Or false if an error occured
preg_match_all("|<[^>]+>(.*)</[^>]+>|U", "<b>example: </b><div align=left>this is a test</div>", $out, PREG_PATTERN_ORDER); // $out[0] is an array of the matched results of the WHOLE pattern echo $out[0][0] . ", " . $out[0][1] . "\n"; // <b>example: </b>, <div align=left>this is a test</div> // $out[1] is an array of the matched results of the first parenthesized pattern echo $out[1][0] . ", " . $out[1][1] . "\n"; // example: , this is a test
1.20.5.5. Split a multilined string by blank line
$array = preg_split("#\n\s*\n#Uis", $text);
1.20.5.6. Remove HTML tag
- Refer to remove spaces between HTML start and end tags
- email:spaces between html tags
// Remove HTML tags $tags = ['noscript', 'p']; return preg_replace('@<('. implode('|', $tags) .')[^>]*?>.*?</\\1>@si', '',$html); // Remove an attribute from string // e.g. remove style attribute $output = preg_replace('/(<[^>]+) style=".*?"/si', '$1', $input);
1.20.5.7. Separate Street Name from Civic Number in Address String
$addresses = [ '100 Baker Street', // => '100 ' '109 - 111 Wharfside Street', // => '109 - 111 ' '40-42 Parkway', // => '40-42 ' '25b-26 Sun Street', // '25b-26 ' '43a Garden Walk', // '43' '6/7 Marine Road', // '6/7 ' '10 - 12 Acacia Ave', // '10 - 12 ' '4513 3RD STREET CIRCLE WEST', // '4513 3' '0 1/2 Fifth Avenue', // '0 1/2 ' '194-03 1/2 50th Avenue', // '194-03 1/2 ' ]; $re = '/^\d+\w*\s*(?:[\-\/]?\s*)?\d*\s*\d+\/?\s*\d*\s*/'; foreach ( $addresses as $k => $v ) { $matches = []; preg_match( $re, $v, $matches ); var_dump( $matches ); }
1.20.6. Read string line by line
foreach (preg_split("/((\r?\n)|(\r\n?))/", $map) as $_line) { $line = explode(',', $_line); // comma separated foreach (explode('|', $line[1]) as $prov) { $r[$prov] = $line[0]; } }
1.20.7. String Ellipsis, Trim php:ellipsis
1.20.8. Encode HTML
- d7:functions:check_plain is
htmlspecialchars($text, ENT_QUOTES, 'UTF-8');- escape both single and double quotes
- sanitize HTML attribute values (pure text)
JavaScript inside an HTML attribute
// Title should be plain text $title = "<span style=\"color: red;\">It's supposed to be, `well`, \"plain\" text</span>"; $flags = ENT_QUOTES; $titleEncoded = htmlspecialchars($title, $flags); // Tips: Which means we can stack usages of htmlspecialchars()! // A double quote " // - htmlspecialchars first time => " // - htmlspecialchars second time => &quot; // - htmlspecialchars third time => &amp;quot; $html = <<<HTML <p>$titleEncoded:</p> <code>{"key": [{"fieldA": "#1"}, {"fieldB": "#2"}]}</code> HTML; $htmlJsonEscaped = htmlspecialchars(json_encode($html), $flags); $jsonEscapedInScript = json_encode($html, JSON_UNESCAPED_UNICODE); $content4 = htmlspecialchars(json_encode("I'm good, too"), $flags); $link = '<a href="#">Click here</a>'; $data = [ 'abc' => 123, 'a' => [ "I'm Li", $link, // no need to escape per value htmlspecialchars($link), ], ]; $dataJsonHtmlString = htmlspecialchars(json_encode(json_encode($data))); $dataJson = htmlspecialchars(json_encode($data)); // Use JavaScript literal for multi-line string $fn = function ($d = '') { return $d; }; $jsFunc = <<<JS console.log({$fn(json_encode($html))}, {$fn(json_encode($data))}); JS; var_dump(<<<HTML <span onmouseover="{$fn(htmlspecialchars($jsFunc))}">Hover me!</span> HTML ); echo <<<HTML <script src="https://cdn.jsdelivr.net/npm/lodash@4.17.21/lodash.min.js"></script> <script> function htmlDecode(input) { var doc = new DOMParser().parseFromString(input, "text/html"); return doc.documentElement.textContent; } function myFunc(content, content2, content3, content4, jsonDataString, jsonData) { console.log('content', content); document.getElementById('content').innerHTML = content; console.log('content2', content2); console.log(content3); console.log(content4); console.log(htmlDecode(content)); console.log($jsonEscapedInScript); console.log(_.isEqual(JSON.parse(jsonDataString), jsonData)); var link = '<a href="#">Click here</a>'; console.log(jsonData.a[1] === link); console.log(jsonData.a[2]); console.log(htmlDecode(jsonData.a[2]) === link); } </script> <span onmouseover="myFunc($htmlJsonEscaped, 123, 'hello I\'m good', $content4, $dataJsonHtmlString, $dataJson);">Hover me!</span> <div id="content"></div> HTML;
sanitize HTML attribute with JavaScript
$fileNameDisplay = htmlspecialchars($fileNameDisplay); $fileNameDisplayAttribute = addslashes($fileNameDisplay); $items[] = <<<HTML <div class="listitem"> <a onclick="picklookup('{$fileNameDisplayAttribute}','{$fileId}');">{$fileNameDisplay}</a> </div> HTML;
- sanitize
$_POSTvalue to safe html text for display
- What it does?
function htmlspecialchars(string $string, int $flags = ENT_QUOTES|ENT_SUBSTITUTE, ?string $encoding = null, bool $double_encode = true)- PHP default
&", unlessENT_NOQUOTESis set'(for ENT_HTML401) or'(forENT_XML1,ENT_XHTMLorENT_HTML5), but only whenENT_QUOTESis set<>
htmlspecialcharsproduces less code thanhtmlentities// Drupal's check_plain if (!function_exists("check_plain")) { function check_plain($text) { return htmlspecialchars($text, ENT_QUOTES, 'UTF-8'); } } echo htmlentities('<Il était une fois un être>.'); // Output: <Il était une fois un être>. // ^^^^^^^^ ^^^^^^^ echo htmlspecialchars('<Il était une fois un être>.'); // Output: <Il était une fois un être>. // ^ ^
htmlentitiesis identical tohtmlspecialchars(). It also converts characters which have html:character entity equivalents
- Flags
ENT_QUOTES- convert both double and single quotes
htmlspecialchars($text, ENT_QUOTES | ENT_XML1, 'UTF-8')- escape values for XML attributes and element data (text node)
- It's ok to have double quotes and single quotes unescaped in element data but escaping is a good practice
- https://www.liquid-technologies.com/Reference/Glossary/XML_EscapingData.html
ENT_SUBSTITUTE- if the text contains an invalid character, those characters are replaced with � characters, instead of returning a completely empty string.
- (no term)
- Default flags
ENT_COMPAT | ENT_HTML401- default flags before PHP 8.1
ENT_COMPAT- convert double quotes only and leave single-quotes alone
- Default from PHP 8.1
ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML401- which is the same as
ENT_QUOTES | ENT_SUBSTITUTE - Single quotes are also escaped
- which is the same as
1.20.9. URL Encode php:urlencode
1.20.11. UTF-8 encoding
mb_convert_encoding(array|string $string, string $to_encoding, array|string|null $from_encoding = null): array|string|false
- Make sure the string is UTF-8 encoded
- after php:file_get_contents
- Or any binary data
// Auto detect source encoding and specify target encoding $twitterHTML = mb_convert_encoding($twitterHTML, 'UTF-8'); $twitterHTML = mb_convert_encoding($twitterHTML, 'UTF-8', mb_detect_encoding($twitterHTML) ); echo $twitterHTML;
1.20.12. JSON
- Input variable is
- String
- result is wrapped with double quotes
- with the following characters escaped by
\"e.g.
</script>becomes<\/script>- Perfect for assigning a PHP string variable to JavaScript variable
<?php $html = <<<HTML <p></p> HTML; $html = json_encode($html); ?> <script> var a = <?php echo $html; ?>; var html = []; html.push(<?php echo $html; ?>); </script>
- with the following characters escaped by
- Number
- result is not wrapped with double quotes
- Array
result is not wrapped with double quotes. The keys are double quoted with no escape
<?php $aNumericArray = json_encode([ 1, "Hello I'm Li. \"", 'Hello I\'m Li. "' ]); var_dump($aNumericArray); // string with no double quotes wrapped // [1,"Hello I'm Li. \"","Hello I'm Li. \""] $aNumericArrayWithForceObject = json_encode([ 1, "Hello I'm Li. \"", 'Hello I\'m Li. "' ], JSON_FORCE_OBJECT); $aNumericArrayWithPrettyPrint = json_encode([ 1, "Hello I'm Li. \"", 'Hello I\'m Li. "' ], JSON_PRETTY_PRINT); var_dump($aNumericArrayWithForceObject); // string with no double quotes wrapped // {"0":1,"1":"Hello I'm Li. \"","2":"Hello I'm Li. \""} var_dump($aNumericArrayWithPrettyPrint, '$aNumericArrayWithPrettyPrint'); // string with no double quotes wrapped // same as $aNumericArray but with indentation $anAssociativeArray = json_encode([ 'a' => 123, 'b' => "Hello I'm Li. \"", 'c' => 'Hello I\'m Li. "', 'd' => ['a', 'b'], ]); var_dump($anAssociativeArray); // string with no quotes wrapped // {"a":123,"b":"Hello I'm Li. \"","c":"Hello I'm Li. \"","d":["a","b"]} ?> <script> var anArray = <?php echo $aNumericArray; ?>; // a JavaScript array variable console.log(anObj1); var anObj = <?php echo $aNumericArrayWithForceObject; ?>; // a JavaScript object variable console.log(anObj); anObj = <?php echo $anAssociativeArray; ?>; // a JavaScript object variable console.log(anObj); </script>
- (no term)
- Object
- By default, the following properties are included
- Explicitly public declared properties
- Dynamically assigned properties
- By default, the following properties are included
- See js:innerHTML
1.20.12.1. UTF-8
$string = 'Svrček'; var_dump($test = mb_detect_encoding($string), $test === 'UTF-8'); // detect if it's UTF-8 using preg_match var_dump($test = preg_match('//u', $string), $test === 1); var_dump($test = json_encode($string), $test === '"Svr\u010dek"'); var_dump($test = json_decode(json_encode($string)), $test === $string); var_dump($test = json_encode($string, JSON_UNESCAPED_UNICODE), $test === '"' . $string . '"'); var_dump($test = json_decode(json_encode($string, JSON_UNESCAPED_UNICODE)), $test === $string); var_dump($test = urlencode(json_encode($string, JSON_UNESCAPED_UNICODE)), $test === '%22Svr%C4%8Dek%22'); var_dump($test = urldecode(urlencode(json_encode($string, JSON_UNESCAPED_UNICODE))), $test === '"' . $string . '"'); var_dump($test = json_decode(urldecode(urlencode(json_encode($string, JSON_UNESCAPED_UNICODE)))), $test === $string);
1.20.12.2. Debug
Use json_last_error(): int and json_last_error_msg(): string to debug
- JSON_ERROR_UTF8 - Malformed UTF-8 characters, possibly incorrectly encoded
- Make sure there's no binary data or non UTF-8 characters in any keys or values in
json_encode($array)- Use 1.20.11
- Make sure there's no binary data or non UTF-8 characters in any keys or values in
1.20.12.3. If a string is a JSON string
<?php $string = 'abc'; $number = 123; $numberAsString = '123'; $zero = 0; $zeroAsString = '0'; $zeroAsFloat = 0.00; $zeroAsFloatInString = '0.00'; $null = null; $array = array('ABC', 'xyz'); function isJSONString($string) { return (1 || @json_decode($string, 1)) && json_last_error() === JSON_ERROR_NONE; } var_dump(JSON_ERROR_NONE === 0); var_dump(json_decode($string, 1) === null); var_dump(json_last_error() !== JSON_ERROR_NONE); var_dump(isJSONString($string) === false); var_dump(json_decode($number, 1) === $number); var_dump(json_last_error() === JSON_ERROR_NONE); var_dump(isJSONString($number) === true); var_dump(json_decode($numberAsString, 1) === $number); var_dump(json_last_error() === JSON_ERROR_NONE); var_dump(isJSONString($numberAsString) === true); var_dump(json_decode($zero, 1) === $zero); var_dump(json_last_error() === JSON_ERROR_NONE); var_dump(isJSONString($zero) === true); var_dump(json_decode($zeroAsString, 1) === $zero); var_dump(json_last_error() === JSON_ERROR_NONE); var_dump(isJSONString($zeroAsString) === true); var_dump(json_decode($zeroAsFloat, 1) === $zero); var_dump(json_last_error() === JSON_ERROR_NONE); var_dump(isJSONString($zeroAsString) === true); var_dump(json_decode($zeroAsFloatInString, 1) === $zeroAsFloat); var_dump(json_last_error() === JSON_ERROR_NONE); var_dump(isJSONString($zeroAsFloatInString) === true); var_dump(json_decode($null, 1) === $null); var_dump(json_last_error() !== JSON_ERROR_NONE); var_dump(isJSONString($null) === false); var_dump(json_decode(json_encode($array), 1) === $array); var_dump(json_last_error() === JSON_ERROR_NONE); var_dump(isJSONString($null) === true);
1.20.12.4. JSON Schema
- https://json-schema.org/
application/schema+jsonorapplication/schema-instance+jsonor justapplication/json
- Keyword: type
- Data Model
- https://json-schema.org/draft/2020-12/json-schema-core.html
- (no term)
- null
- boolean
- object
- (no term)
array
{ "my_property:": { "type": "array", "items": {"type": "string"} } }- number
- base-10 decimal number. The following types are accepted but there's no distinction
- integer
- string
- (no term)
"type": "string"or"type": ["string", "null"]
- Keyword: enum
{ "type": "integer", "enum": [1, 2, 3] }{ "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "product": { "type": [ "boolean", "string" ], "enum": [ "off", "on", "semi-on", true, false ] } } } - Keyword: if then else
{ "type": "object", "properties": { "street_address": { "type": "string" }, "country": { "default": "United States of America", "enum": ["United States of America", "Canada"] } }, "if": { "properties": { "country": { "const": "United States of America" } } }, "then": { "properties": { "postal_code": { "pattern": "[0-9]{5}(-[0-9]{4})?" } } }, "else": { "properties": { "postal_code": { "pattern": "[A-Z][0-9][A-Z] [0-9][A-Z][0-9]" } } } } - Keyword: allOf
{ "allOf": [ { "if": { ... }, "then": { ... } }, { "if": { ... }, "then": { ... } } ] }
1.20.13. Back slashes for quotes and other characters
- Add backslashes before the following characters
- Single quote
- double quote
- backslash
- NUL (the NUL byte)
$str = "Is your name O'Reilly?"; var_dump(addslashes($str) === "Is your name O\\'Reilly?"); // Is your name O\'Reilly? var_dump(stripslashes($str) === $str); var_dump(stripslashes(addslashes($str)) === $str);
1.20.14. Replace php:str_replace
- Works with multibyte
- Search and replace all from left to right
- use
str_ireplace()for case-insensitive - Binary safe
- Parameters
- $count
- it's not for limit but instead the count of replacements
str_replace( mixed $search, mixed $replace, mixed $subject [, int &$count ] ) : mixed
// single search and replace str_replace( 'a', 'b', 'abc'); // multiple search and replace :: change a to b, x to y str_replace( ['a','x'], ['b', 'y'], 'abcxyz'); // when $subject is an array, search and replace is performed with every entry of subject and return value is an array var_export(str_ireplace(['n/a'], '', "N/A hello world n/a") === ' hello world '); var_export(str_ireplace(['n/a'], '', ["N/A hello world n/a", "Hello World n/a N/A"]) === [" hello world ", "Hello World "]);
1.20.15. Replace first or last occurance
// substr_replace ($string, $replacement, $start, $length = null) : string|string[] function str_flreplace($search, $replace, $subject, $isFirstReplace = 1) { $pos = strpos($subject, $search); if ( ! $isFirstReplace) { // strrpos :: not strpos! Find the position of the last occurrence of a substring in a string $pos = strrpos($subject, $search); } if ($pos !== false) { $subject = substr_replace($subject, $replace, $pos, strlen($search)); } return $subject; } function str_freplace($search, $replace, $subject) { $search = '/' . preg_quote($search, '/') . '/'; return preg_replace($search, $replace, $subject, 1); } $search = '?'; $replace = 123; $subject = "SELECT * FROM loans WHERE loanid = ? AND company = ?"; $a = str_flreplace($search, $replace ,$subject); var_dump('$a', $a); // "SELECT * FROM loans WHERE loanid = 123 AND company = ?" $b = str_flreplace($search, $replace, $subject, false); // "SELECT * FROM loans WHERE loanid = ? AND company = 123" var_dump('$b', $b); $c = str_freplace($search, $replace, $subject); // "SELECT * FROM loans WHERE loanid = ? AND company = 123" var_dump('$c', $c);
1.20.16. Find the position of the first/last occurrence of a substring in a string
- Last occurance
strrpos(string $haystack, string $needle, int $offset = 0): int|false- case-insensitve
strripos(string $haystack, string $needle, int $offset = 0): int|false
- First occurance
strpos(string $haystack, string $needle, int $offset = 0): int|false- case-insensitive
stripos(string $haystack, string $needle, int $offset = 0): int|false
1.20.17. First character uppercase/lowercase - ucfirst() lcfirst()
1.20.18. First letter of each word uppercase ucwords()
ucwords(string $string, string $separators = " \t\r\n\f\v"): string
1.20.19. Start with / End with
$prefix = '/inventory/'; // case-insensitive compare if( strcasecmp(substr($request_uri, 0, strlen($prefix)), $prefix) === 0 ){ // it has that prefix and remove the prefix // $new_uri = substr($request_uri, strlen($prefix)); } // case-sensitive compare if ( substr($request_uri, 0, strlen($prefix)) === $prefix ) {} function endsWith($haystack, $needle) { $length = strlen($needle); return $length > 0 ? substr($haystack, -$length) === $needle : true; } // one liner end with $haystack = 'path/to/file'; $needle = 'to/file'; if (strlen($needle) > 0 ? substr($haystack, -strlen($needle)) === $needle : true) {}
1.20.20. sprintf and printf
- See 1.20.21 for named arguments
sprintf(string $format, mixed ...$values): string
// Syntax %[flags][width][.precision]specifier // escape % with %% $num = 5; $location = 'tree'; $format = 'There are %d monkeys in the %s'; echo sprintf($format, $num, $location); // `n$` is a specifier and when it exists, it must be immediately after % $format = 'The %2$s contains %1$d monkeys'; // The tree contains 5 monkeys $format = 'The %2$s contains %1$04d monkeys'; // The tree contains 0005 monkeys // `'.` is a flag, it pads the result with the character `'(char)` echo sprintf("%'.9d\n", 123); // ......123 // I don't know.. It's the same as `echo sprintf("%09d\n", 123);` echo sprintf("%'.09d\n", 123); // 000000123 // decimal points echo sprintf("%.5f", 123); // 123.00000
1.20.21. strtr
// byte-to-byte strtr(string $string, string $from, string $to): string // The longest keys will be tried first. Once a substring has been replaced, its new value will not be searched again strtr(string $string, array $replace_pairs): string
$trans = array("h" => "-", "hello" => "hi", "hi" => "hello"); // `h` is not picked because there're longer matches and replaced text was not searched again strtr("hi all, I said hello", $trans) === 'hello all, I said hi'; $strParams = [ ':name' => 'Dave123', 'Dave123' => ':name2 or :password', // Not replaced because it's not found in the original source text ':name2AndPassword' => ':name2 or :password', ':name2' => 'Steve', ':pass' => '7hf2348', ]; var_dump(strtr("My name is :name, not :name2. :name2AndPassword", $strParams) === "My name is Dave123, not Steve. :name2 or :password");
1.20.22. Random base64 string
$str = substr(base64_encode(sha1(mt_rand())), 0, 16);
1.20.23. Hash: md5, sha1
- Hashed string (checksum) is a one-way transformation on the original string. Another string has a different checksum
1.20.23.1. hash()
Syntax
hash( string $algo, string $data, bool $binary = false, array $options = [] ): stringprint_r(hash_algos());
sha1($text) === hash('sha1', $text)
1.20.23.2. sha1()
- Unsuitable for storing passwords
Result is a 40 lowercase hexadecimal characters (160 bits or 20 bytes).
$binary = truereturns binary of length 20. Maximum message size is 264-1 bitssha1(string $string, bool $binary = false): string
- numbers 0-9 and a-f (base 16 = 2^4). 1 hexadeciaml digit is 4 bit.
- 9
- decimal 9
- a
- decimal 10
- b
- decimal 11
- c
- decimal 12
- d
- decimal 13
- e
- decimal 14
- f
- decimal 15
- 11
- decimal 16
1.20.23.3. md5
- Unsuitable for storing passwords
- Output is fixed-length 128 bits = 16 bytes = 32 hexadecimal digits/numbers
<?php // Signed cookie example function sign_string($string) { // Using $salt makes it hard to guess how $checksum is generated // Caution: changing salt will invalidate all signed strings $salt = "Simple salt"; $checksum = sha1($string . $salt); // Any hash algorithm would work // return the string with the checksum at the end return $string . '--' . $checksum; } function signed_string_is_valid($signed_string) { $array = explode('--', $signed_string); if (count($array) != 2) { // string is malformed or not signed return false; } // Sign the string portion again. Should create same // checksum and therefore the same signed string. $new_signed_string = sign_string($array[0]); if ($new_signed_string == $signed_string) { return true; } else { return false; } } // Uncomment to demonstrate usage $string = "This is a test."; echo "Original: " . $string . "<br />"; echo "<br />"; $signed_string = sign_string($string); echo "Signed: " . $signed_string . "<br />"; echo "<br />"; $valid = signed_string_is_valid($signed_string); echo "Valid? " . ($valid ? 'true' : 'false') . "<br />"; echo "<br />"; $tampered_string = "This was a test.--521d61df5d4c239a1f5a191ff3803826b60587a9"; echo "Tampered: " . $tampered_string . "<br />"; echo "<br />"; $valid = signed_string_is_valid($tampered_string); echo "Valid? " . ($valid ? 'true' : 'false') . "<br />"; echo "<br />";
1.20.23.4. hash_hmac
hash_hmac(
string $algo,
string $data,
string $key,
bool $binary = false
): string
- Generate a keyed hash value using HMAC method
- MAC (Message Authentication Code) is produced from a message and a secret key by a MAC algo.
- It's impossible to produce the MAC of a message and a secret key without knowing the secret key
- A MAC of the same message produced by a different key looks unrelated
- Even knowing the MAC of other messages does not help in computing the MAC of a new message
- Example, our server receives a long message and a MAC, the server uses the message and the secret key to compute the MAC and finds it's the same as the provided MAC, then the server sees this message comes from a valid source. A MAC is also called a signature based on secret key.
- HMAC is a MAC based on a hash function
- There're other ways to define a MAC e.g. MAC algo based on block ciphers such as CMAC
hash_hmac('md5', 'The quick brown fox jumps over the lazy dog', 'key'); // 32 hexadecimal hash
1.20.24. Password Hash
if ( isset( $_POST['submit'] ) ) { $password = $_POST['password']; echo "Text password: " . $password . "<br />"; // Encryption example $hashed_password = password_hash( $password, PASSWORD_BCRYPT ); // default PASSWORD_DEFAULT now actually uses PASSWORD_BCRYPT, should stick with default as // PHP will upgrade the underlying algo in the future echo "Hashed password: " . $hashed_password . "<br />"; echo "<br />"; // Verification example $wrong_password = "anything_else"; $is_match = password_verify( $wrong_password, $hashed_password ); echo "Attempt 1: " . ( $is_match ? 'correct' : 'wrong' ) . "<br />"; $is_match = password_verify( $password, $hashed_password ); echo "Attempt 2: " . ( $is_match ? 'correct' : 'wrong' ) . "<br />"; echo "<br />"; echo "<hr />"; // rehash $opt = ['cost' => 12 ]; if (password_needs_rehash($password, PASSWORD_DEFAULT, $opt)) { $newHash = password_hash($password, PASSWORD_DEFAULT, $opt); } }
1.20.25. encrypt, decrypt
- should be used instead of
Encrypt
$data = "hello"; $method = "aes-256-cbc"; $key = "123455"; $iv = "1234567890123456"; // has to be 16 bytes or 16 text characters $iv = openssl_random_pseudo_bytes(16); // $iv should be different every time $options = 0; // If $data, $key and $iv are normal text like above, 0 can be used. Output is base64 encoded string // Otherwise use OPENSSL_RAW_DATA. Output is binary $encryptedMessage = openssl_encrypt($data, $method, $key, $options, $iv); echo $encryptedMessage.PHP_EOL; // because $options is 0, the encrypted message is base64 encoded $decryptedMessage = openssl_decrypt($encryptedMessage, $method, $key, $options, $iv);
Decrypt
$fromNode = 'encryptedString$intializedVector'; $fromNode = explode('$', $fromNode); $key = '64 hex characters'; // same as in nodejs $data = hex2bin($fromNode[0]); $keyAES = hex2bin($key); $iv = hex2bin($fromNode[1]); $options = OPENSSL_RAW_DATA; // Input are all binary, output is binary, too. // But because the encrypted message is a string, the decrypted message is a string. $decrypted = openssl_decrypt($data, 'aes-256-cbc', $keyAES, $options, $iv); // If $data, $keyAES and $iv are raw data (bytes), use OPENSSL_RAW_DATA as $options // if all 3 of them are string, use 0 as $options
- If
$encryptedMessage,$keyand$ivare all hex, you need to convert them to binary to decrypt - Refer to nodejs:decrypt
- If
1.20.26. Crypt
crypt(string $string, string $salt): string
- It creates a one-way hashed string and it is not yet binary safe
- $salt
- Based on the salt provided, it will create a hash using the corresponding algorithm
- Can be omitted but not recommended. When omitted, PHP will auto-generate a 12 character salt using MD5 algo if it's available, if not, a 2 character salt using DES algo
- 2 character salt from the alphabet
./0-9A-Za-z - 9 character salt starting with
_followed by 4-character-iteration-count and 4-character-salt - 12 character salt starting with
$1$ - CRYPT_BLOWFISH
- Waring !!
- using this algo, the
$stringis truncated to 72 bytes!
- 16 character salt prefixed with
$5$, may be followed byrounds=<N>$whereNis used to indicate how many times the hashing loop should be executed, default is 5000, with minimum 1000 and maximum of 999999999 - same as CRYPT_SHA256 but prefix is
$6$
- Based on the salt provided, it will create a hash using the corresponding algorithm
$user_input = 'rasmuslerdorf'; $hashed_password_stored_in_db = crypt($user_input, '$5$rounds=5000$usesomesillystringforsalt$'); // Only the first 16 characters of the salt will be used: usesomesillystri $hashed_password_stored_in_db = '$5$rounds=5000$usesomesillystri$KqJWpanXZHKq2BOB43TSaYhEWsQ1Lr5QNyPCDH/Tp.6'; // KqJWpanXZHKq2BOB43TSaYhEWsQ1Lr5QNyPCDH/Tp.6 is the hashed result var_dump(hash_equals($hashed_password_stored_in_db, crypt($user_input, $hashed_password_stored_in_db)));
1.20.27. Random
function randomString(int $length, $options = array()): string { if ($options['intOnly'] ?? null) { return strval(random_int(pow(10, $length - 1), pow(10, $length) - 1)); } $returnBase64 = function($rand) use ($length, $options) { $r = substr(base64_encode($rand), 0, $length); return ($options['lowerCase'] ?? null) ? strtolower($r) : $r; }; if (is_callable('random_bytes')) { return $returnBase64(random_bytes($length * 2)); } if (is_callable('openssl_random_pseudo_bytes')) { return $returnBase64(openssl_random_pseudo_bytes($length * 2)); } return $returnBase64(mt_rand()); }
1.20.28. True or False on string
// index.php? var_dump($_GET['notexist']); // null // index.php?abc= var_dump($_GET['abc']); // "" empty string // So $_GET['abc'] can be a string or null var_dump(trim(null)); // "" empty string // trim($_GET['abc']) returns only string $b = ''; var_dump(($b) ? true : false); // false $a = [ '' => false, '0' => false, '1' => true, '-1' => true, ];
1.20.29. If string is an integer
// Option #1. // Limitation: leading zero string e.g. '05' is not considered an integer if ($value === strval(intval($value))) { return intval($value); } var_dump(filter_var('2', FILTER_VALIDATE_INT)); // 2 var_dump(filter_var('2.0', FILTER_VALIDATE_INT)); // false var_dump(filter_var('2.1', FILTER_VALIDATE_INT)); // false var_dump(filter_var(2, FILTER_VALIDATE_INT)); // 2 var_dump(filter_var(2.0, FILTER_VALIDATE_INT)); // 2 var_dump(filter_var(2.1, FILTER_VALIDATE_INT)); // false
1.21. Conversion to boolean
/* * https://www.php.net/manual/en/language.types.boolean.php */ var_dump(0 == false); var_dump(0.0 == false); var_dump(-0.0 == false); var_dump('0' == false); var_dump('' == false); var_dump([] == false); var_dump(null == false); // SimpleXML objects created from attributeless empty elements, i.e. elements which have neither children nor attributes.
1.22. URL
1.22.1. urlencode() :: encode the query string (the part after ?)
- encodes spaces as plus signs
- eq. to js:encodeURIComponent
- Should be used to encode keys and values pairs of POST data
$query_string = 'foo=' . urlencode($foo) . '&bar=' . urlencode($bar); echo '<a href="mycgi?' . htmlentities($query_string) . '">';
1.22.2. rawurlencode
- encode path (the part before
?). Almost the same asurlencodeexcept it enocodes space as%20 Basics
echo '<a href="ftp://user:', rawurlencode('foo @+%/'), '@ftp.example.com/x.txt">'; echo '<a href="http://example.com/department_list_script/', rawurlencode('sales and marketing/Miami'), '">';
Use to encode filename in header:content-disposition
function fileNameForHeader( $fileName = '' ) { // Usage: header("Content-disposition: attachment; filename=\"fileNameForHeader('FileName')\""); $fileName = str_replace( array( "\n", "\r", "\r\n", ' ', ',' ), '_', $fileName ); $fileName = str_replace( array( "'", '"' ), '', $fileName ); return rawurlencode( $fileName ); }
1.22.3. http_build_query
- Return a URL-encoded query string
- By default perfect for cURL POST content-type
application/x-www-form-urlencoded- spaces are encoded as
+which is eq. tourlencode - Also good for GET url query string
- spaces are encoded as
- By default perfect for cURL POST content-type
var_dump(http_build_query([
'a' => 'hello world', // a=hello+world
'b' => ' hello world ', // &b=+hello+world+
'c' => [1, 2, 3], // &c%5B0%5D=1&c%5B1%5D=2&c%5B2%5D=3 which is &c[0]=1&c[1]=2&c[2]=3
'd' => ['1' => 'hello', '2' => 'world'], // &d%5B1%5D=hello&d%5B2%5D=world which is &d[1]=hello&d[2]=world
'e' => ['1' => ' hello ', '2' => 'wor ld'], // &e%5B1%5D=+hello+&e%5B2%5D=wor+ld which is &e[1]=+hello+&e[2]=wor+ld
'f' => ['hello' => 'world', 'hello2' => 'world'], // &f%5Bhello%5D=world&f%5Bhello2%5D=world which is &f[hello]=world&f[hello2]=world
'g' => [' hello ' => ' world ', 'hello 2' => 'wor ld'], // &g%5B+hello+%5D=+world+&g%5Bhello+2%5D=wor+ld which is &g[+hello+]=+world+&g[hello+2]=wor+ld
'h' => json_encode([' hello ' => ' world ', 'hello 2' => 'wor ld']), // &h=%7B%22+hello+%22%3A%22+world+%22%2C%22hello+2%22%3A%22wor+ld%22%7D which is &h={"+hello+":"+world+","hello+2":"wor+ld"}
' hello ' => [' hello ' => ' world ', 'hello 2' => 'wor ld'], // &+hello+%5B+hello+%5D=+world+&+hello+%5Bhello+2%5D=wor+ld which is &+hello+[+hello+]=+world+&+hello+[hello+2]=wor+ld
'wor ld' => [' hello ' => ' world ', 'hello 2' => 'wor ld'], // &wor+ld%5B+hello+%5D=+world+&wor+ld%5Bhello+2%5D=wor+ld which is &wor+ld[+hello+]=+world+&wor+ld[hello+2]=wor+ld
]));
1.22.4. parse_str :: parse query parameter value as an array
$url = $_SERVER['REQUEST_URI']; $parts = parse_url($url); if ($parts !== false) { // if there's error, it will return false parse_str($parts['query'], $query); echo $query['email']; } // parse_str ($str, array &$result = null) : void $str = "first=value&arr[]=foo+bar&arr[]=baz"; parse_str($str, $output); echo $output['first']; // value echo $output['arr'][0]; // foo bar echo $output['arr'][1]; // baz function removeOneQueryParameter($url, $queryParameter) { if (($urlParsed = parse_url($url)) === false) { return $url; } parse_str($urlParsed['query'], $query); unset($query[$queryParameter]); $queryPart = http_build_query($query); return strtok($url, '?') . ($queryPart ? "?{$queryPart}" : ''); } var_dump(removeOneQueryParameter('https://abc.com?abc=123', 'abc') === 'https://abc.com'); var_dump(removeOneQueryParameter('https://abc.com?xyz=&abc=123', 'abc') === 'https://abc.com?xyz=');
1.22.5. Current URL path. See php:parse_url
$path_only = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);
1.22.6. The last component of path
basename("/etc/passwd") // passwd basename("/etc/sudoers.d") // sudoers.d basename("/etc/sudoers.d", ".d") // sudoers
1.22.7. Add URL Parameter
if (!function_exists('http_build_url')) { /** * URL constants as defined in the PHP Manual under "Constants usable with * http_build_url()". * * https://github.com/jakeasmith/http_build_url/blob/master/src/http_build_url.php * @see http://us2.php.net/manual/en/http.constants.php#http.constants.url */ if (!defined('HTTP_URL_REPLACE')) { define('HTTP_URL_REPLACE', 1); } if (!defined('HTTP_URL_JOIN_PATH')) { define('HTTP_URL_JOIN_PATH', 2); } if (!defined('HTTP_URL_JOIN_QUERY')) { define('HTTP_URL_JOIN_QUERY', 4); } if (!defined('HTTP_URL_STRIP_USER')) { define('HTTP_URL_STRIP_USER', 8); } if (!defined('HTTP_URL_STRIP_PASS')) { define('HTTP_URL_STRIP_PASS', 16); } if (!defined('HTTP_URL_STRIP_AUTH')) { define('HTTP_URL_STRIP_AUTH', 32); } if (!defined('HTTP_URL_STRIP_PORT')) { define('HTTP_URL_STRIP_PORT', 64); } if (!defined('HTTP_URL_STRIP_PATH')) { define('HTTP_URL_STRIP_PATH', 128); } if (!defined('HTTP_URL_STRIP_QUERY')) { define('HTTP_URL_STRIP_QUERY', 256); } if (!defined('HTTP_URL_STRIP_FRAGMENT')) { define('HTTP_URL_STRIP_FRAGMENT', 512); } if (!defined('HTTP_URL_STRIP_ALL')) { define('HTTP_URL_STRIP_ALL', 1024); } /** * Build a URL. * * The parts of the second URL will be merged into the first according to * the flags argument. * * @param mixed $url (part(s) of) an URL in form of a string or * associative array like parse_url() returns * @param mixed $parts same as the first argument * @param int $flags a bitmask of binary or'ed HTTP_URL constants; * HTTP_URL_REPLACE is the default * @param array $new_url if set, it will be filled with the parts of the * composed url like parse_url() would return * * @return string */ function http_build_url($url, $parts = [], $flags = HTTP_URL_REPLACE, &$new_url = []) { is_array($url) || $url = parse_url($url); is_array($parts) || $parts = parse_url($parts); isset($url['query']) && is_string($url['query']) || $url['query'] = NULL; isset($parts['query']) && is_string($parts['query']) || $parts['query'] = NULL; $keys = ['user', 'pass', 'port', 'path', 'query', 'fragment']; // HTTP_URL_STRIP_ALL and HTTP_URL_STRIP_AUTH cover several other flags. if ($flags & HTTP_URL_STRIP_ALL) { $flags |= HTTP_URL_STRIP_USER | HTTP_URL_STRIP_PASS | HTTP_URL_STRIP_PORT | HTTP_URL_STRIP_PATH | HTTP_URL_STRIP_QUERY | HTTP_URL_STRIP_FRAGMENT; } elseif ($flags & HTTP_URL_STRIP_AUTH) { $flags |= HTTP_URL_STRIP_USER | HTTP_URL_STRIP_PASS; } // Schema and host are alwasy replaced foreach (['scheme', 'host'] as $part) { if (isset($parts[$part])) { $url[$part] = $parts[$part]; } } if ($flags & HTTP_URL_REPLACE) { foreach ($keys as $key) { if (isset($parts[$key])) { $url[$key] = $parts[$key]; } } } else { if (isset($parts['path']) && ($flags & HTTP_URL_JOIN_PATH)) { if (isset($url['path']) && substr($parts['path'], 0, 1) !== '/') { // Workaround for trailing slashes $url['path'] .= 'a'; $url['path'] = rtrim( str_replace(basename($url['path']), '', $url['path']), '/' ) . '/' . ltrim($parts['path'], '/'); } else { $url['path'] = $parts['path']; } } if (isset($parts['query']) && ($flags & HTTP_URL_JOIN_QUERY)) { if (isset($url['query'])) { parse_str($url['query'], $url_query); parse_str($parts['query'], $parts_query); $url['query'] = http_build_query( array_replace_recursive( $url_query, $parts_query ) ); } else { $url['query'] = $parts['query']; } } } if (isset($url['path']) && $url['path'] !== '' && substr($url['path'], 0, 1) !== '/') { $url['path'] = '/' . $url['path']; } foreach ($keys as $key) { $strip = 'HTTP_URL_STRIP_' . strtoupper($key); if ($flags & constant($strip)) { unset($url[$key]); } } $parsed_string = ''; if (!empty($url['scheme'])) { $parsed_string .= $url['scheme'] . '://'; } if (!empty($url['user'])) { $parsed_string .= $url['user']; if (isset($url['pass'])) { $parsed_string .= ':' . $url['pass']; } $parsed_string .= '@'; } if (!empty($url['host'])) { $parsed_string .= $url['host']; } if (!empty($url['port'])) { $parsed_string .= ':' . $url['port']; } if (!empty($url['path'])) { $parsed_string .= $url['path']; } if (!empty($url['query'])) { $parsed_string .= '?' . $url['query']; } if (!empty($url['fragment'])) { $parsed_string .= '#' . $url['fragment']; } $new_url = $url; return $parsed_string; } } function _lili_campaign_url( $link, $options = []) { if (!empty($options) && isset($options['utm_campaign'])) { $default = [ 'utm_source' => 'newcom', // default to newcom, $options can override $default 'utm_medium' => 'email', // default to email, $options can override $default // 'utm_campaign' = 'provided', // if utm_campaign is not set, it will return the original link 'utm_content' => date('YmdGis'), // default to current date and time, $options can override $default ]; $options = array_merge( $default, $options ); $url = @parse_url($link); if (!empty($url)) { parse_str($url['query'], $params); // existing url parameters will remain regardless of $default and $options $params = array_merge($options, $params); // move utm_* parameters to the end foreach ($options as $item => $value) { $v = $params[$item]; unset($params[$item]); $params[$item] = $v; } // url parameter values are encoded $url['query'] = http_build_query($params); $link = http_build_url($url); } } return $link; }
1.23. Random Integer Number
There're several ways
$numbers = range(0,12); // 0, 1, 2, ..., 12 shuffle($numbers); echo $numbers[0]; echo(mt_rand(10,100)); // min: 0, max: 100 $rnd = rand(pow(10, 8-1), pow(10, 8)-1); // random 7 digits integer
1.24. HTML encode and decode
$t = '<a href="http://expample.com">Hello L\'automobile from "Me"</a>'; var_dump(htmlentities($t, ENT_QUOTES, 'UTF-8')); var_dump(html_entity_decode($t, ENT_QUOTES, 'UTF-8')); // The same as D7 decode_entities($t)
ENT_QUOTES is to convert both single ' and double " quotes.
1.25. Heredoc, Nowdoc
1.25.1. Heredoc
- Result is a double-quoted string
- Limitation
Can't interpolate constants, class constants and class static properties directly. Use
sprintf,printf, or$fninstead<?php function echoSomething($text) { echo $text; } class A { const NAME = 'abc'; public static $abc = 'xyz'; public static function hello() { $fn = function ($d = '') { return $d; }; $sum = 0; echo <<<TEXT {$fn(self::NAME)} {$fn(self::$abc)} {$fn((function ($abc) { $r = ''; if ($abc) { $r = $abc; } return $r; })(self::$abc))} {$fn((function ($abc) { ob_start(); echoSomething($abc); return ob_get_clean(); })(self::$abc))} {$fn((function ($array) { array_walk($array, function ($v, $k) use (&$r) { $r .= <<<HTML <span>{$k}: {$v}</span> HTML; }, $r = ''); return $r; })(array('an', 'array', 'to walk/loop')))} {$fn( /* Comment out with <?php blocks <?php $a = 1 ?> */ )} {$fn((function ($results, $fn) { $f = function($array) use (&$f, $fn) { return <<<HTML <ul> {$fn((function ($array, $f, $fn) { array_walk($array, function ($v, $k) use (&$r, $f, $fn) { $r .= <<<HTML <li> {$fn(htmlspecialchars($k))} {$fn(is_array($v) ? $f($v) : ': ' . htmlspecialchars($v))} HTML; }, $r = ''); return $r; })($array, $f, $fn))} </ul> HTML; }; return $f($results); })(['a' => ['b' => 'c']], $fn))} {$fn("New Sum is: " . ($sum = $sum + 1) . PHP_EOL)} {$fn("New Sum is: " . $newSum = (function ($sum) { return ++$sum; })($sum))} {$fn((list($newSum, $newSumHTML) = (function ($sum) { return array(++$sum, '<p>HTML</p>'); })($sum)) == '')} {$fn("New Sum is: {$newSum}" . PHP_EOL)} {$fn($newSumHTML)} TEXT; } } A::hello();
Query
$params = []; $query = <<<MYSQL SELECT * FROM myTable WHERE aCol = {$fn( ($params = array_merge($params, array($a))) ? '?' : '' )} bCol = {$fn( ($params[] = [$a]) ? '?' : '' )} MYSQL;
1.25.1.1. Complex (curl) syntax
$juice = 'apple'; $a = array( 'a' => 'abc', ); $referToVariableJuice = 'juice'; function getVariableName() { return 'juice'; } function returnArray($data = null) { return isset($data) ? $data : array('a' => 'abc'); } $lambda = function ($data) { return $data; }; $getNameLambda = function () { return 'getNameLambda'; }; class Foo { const NAME = 'name: foo'; public $bar = 'I am bar.'; public static $staticProperty = 'staticProperty'; } $foo = new Foo(); $bar = 'bar'; $baz = array('foo', 'bar', 'baz', 'quux'); echo <<<HTML Simple: $juice => apple, use \$juice to escape Dollar sign simple syntax: ${a['a']} => abc, use \${a['a']} to escape ${juice} => apple, use \${juice} to escape ${juice}s => apples Complex (curl) syntax: {$a['a']} => abc, use {\$a['a']} to escape {$juice}s => apples, {${$referToVariableJuice}} => ${$referToVariableJuice} => apple Complex (curl) syntax access class properties using variables: {$foo->bar} => I am bar. {$foo->{$baz[1]}} => {\$foo->bar} => {$foo->bar} => I am bar. Complex (curl) syntax can call function! {${getVariableName()}} => \${getVariableName()} (syntax error on phpStorm and PHP 5.4-) => apple => \${juice} {$getNameLambda()} => getNameLambda {$lambda(time())} => Unix timestamp {$lambda(time() / 2)} => Unix timestamp {$lambda(time() . ' ' . $lambda( time() / 2 ))} {$lambda(returnArray()['a'])} => abc {$lambda(returnArray(array('a' => 'xyz'))['a'])} => xyz {$lambda((returnArray()['a']) ? time() : time() / 2)} => Unix timestamp {$lambda(Foo::NAME)} => name: foo {$lambda(Foo::$staticProperty)} => staticProperty PHP v7.0 below {$lambda( call_user_func( function ($delimiter) { $r = ''; foreach (array('hello', 'world') as $i) { $r .= <<<HTML <p>{$i}</p> {$delimiter} HTML; } return $r; }, $delimiter ) )} PHP v7.0+ {$lambda( (function ($delimiter) { $r = ''; foreach (array('hello', 'world') as $i) { $r .= <<<HTML <p>{$i}</p> {$delimiter} HTML; } return $r; })( $delimiter ) )} HTML; // lambda can be nested {$lambda( call_user_func( function ($delimiter, $lambda) { $r = ''; foreach (array('hello', 'world') as $i) { $r .= <<<HTML <p>{$i}</p> {$lambda(date('Y-n-j'))} HTML; } return $r; }, $delimiter, $lambda ) )}
1.25.1.2. Use sprintf
$html = <<<HTML <span>{$obj->name[1]}</span> HTML; // this is compatible with PHP 5.3+ including 5.3 $query = sprintf(<<<MYSQL SELECT * FROM t1 WHERE id IN (%s, %s) MYSQL , $a, $a); // For PHP 7.3+ and including 7.3 $query = sprintf(<<<MYSQL SELECT * FROM t1 WHERE id IN (%s, %s) MYSQL, $a, $a);
$string = <<<heredoc plain text and now a function: %s heredoc; $string = sprintf($string, testfunction());
1.25.2. Nowdoc can't expand variables and the result is single quoted string
echo <<<'EOT' <span>{$obj->name[1]}</span> EOT;
1.25.3. Alternative
- Also provides syntax highlighting and code inspection in PhpStorm
$a = " <div class=\"lookupanchor\"> <div class=\"minilookup\" id=\"{$id}_lookup\"> <a id=\"{$id}_lookup_closer\" class=\"labelbutton closer\" onclick=\"gid('{$id}_lookup').style.display='none';\">x</a> <div id=\"{$id}_lookup_view\" class=\"lookupview\" $styles></div> </div> </div>";
1.25.4. Regex to find closing tag
(?:SQL[\s]+$|SQL[)},]+|^\s+SQL)
1.26. Buffer
http://php.net/manual/en/book.outcontrol.php https://phpfashion.com/everything-about-output-buffering-in-php Send headers to browser after php has begun outputting data.
// set header ob_start(); echo "Hello\n"; setcookie("cookiename", "cookiedata"); // the same header cannot be set because it's already been sent ob_end_flush();
ob_start() can be stacked
Flush means send to output
- ob_end_flush
- send and turn off output buffering
- ob_get_flush
- send, return and turn off
- ob_flush
- send
- ob_clean
- erase the output buffer
- ob_end_clean
- erase and turn off
- ob_get_clean
- return, erase and turn off eq. to ob_get_contents + ob_end_clean
- ob_get_contents
- return
ob_get_length
ob_start(); ?> <div id="site_header"> <div id="site_header_inner"> <div id="site_header_logo"><?php echo $PhpVar; ?></div> <div id="site_header_countdown">BARE XX DAGER IGJEN</div> </div> </div> <?php $output = ob_get_clean(); //ob_flush(); return $output;
function mmm_wpml_shortcode_func(){ ob_start(); do_action('icl_language_selector'); $output_string = ob_get_contents(); ob_end_clean(); return $output_string; }
1.27. Template System
static public function renderHeader($data = array()) { $localData = array(); return self::renderView(__DIR__ . '/../views/header.php', $localData); } static private function renderView($path, $data=array()) { ob_start(); if (file_exists($path)) { include ($path); } // else throw new TemplateNotFoundException(); return ob_get_clean(); }
- header.php
1.28. Image
1.28.1. GD vs Imagick
- GD is the oldest. Imagick might not exist in some hosting
- GD only supports JPG, PNG, GIF, WBMP, WebP, XBM and XPM
- If TIFF needs to be supported, use Imagick
if(extension_loaded('gd')) { print_r(gd_info()); } else { echo 'GD is not available.'; } if(extension_loaded('imagick')) { $imagick = new Imagick(); print_r($imagick->queryFormats()); } else { echo 'ImageMagick is not available.'; }
1.28.2. Create image in memory and output
$data = 'iVBORw0KGgoAAAANSUhEUgAAABwAAAASCAMAAAB/2U7WAAAABl' . 'BMVEUAAAD///+l2Z/dAAAASUlEQVR4XqWQUQoAIAxC2/0vXZDr' . 'EX4IJTRkb7lobNUStXsB0jIXIAMSsQnWlsV+wULF4Avk9fLq2r' . '8a5HSE35Q3eO2XP1A1wQkZSgETvDtKdQAAAABJRU5ErkJggg=='; $data = base64_decode($data); $im = imagecreatefromstring($data); if ($im !== false) { header('Content-Type: image/png'); imagepng($im); imagedestroy($im); } else { echo 'An error occurred.'; }
I think header:cache-control is automatically set to no cache To set Content-Length:
ob_start(); imagejpeg($img); header('Content-Type: image/jpg'); header("Content-length: " . ob_get_length()); // images do not compression e.g. gzip imagedestroy($im); // send it ob_flush();
1.28.3. Resize image
require 'SimpleImage.php'; try { // Create a new SimpleImage object $image = new \claviska\SimpleImage(); //$new_file = __DIR__ . '/'.$file_path.$file_no_ext.'_200x200f.'.$ext; $new_file = $file_path.$file_no_ext.'_200x200f.'.$ext; $image ->fromFile($file_orig_full) // load parrot.jpg //->autoOrient() // adjust orientation based on exif data ->bestFit(200, 200) ->toFile($new_file); // output to the screen $localW = $image->getWidth(); $localH = $image->getHeight(); var_dump($localW, $localH); } catch(Exception $err) { // Handle errors echo $err->getMessage(); }
1.29. Apply XSLT on XML
$xml = new DOMDocument; $xml->load('cdcatalog.xml'); $xsl = new DOMDocument; $xsl->load('cdcatalog.xsl'); $proc = new XSLTProcessor; $proc->importStyleSheet($xsl); echo $proc->transformToXML($xml);
1.30. Block Referral Traffic
if ( isset( $_SERVER['HTTP_REFERER'] ) ) { $_referer_domain = array( "~timebie\.com~i", "~ana-white\.com~i", "~mlizcochico\.com~i", "~adishofdailylife\.com~i", "~techvibes\.com~i", "~sociableblog\.com~i", "~laurenconrad\.com~i", "~gingerhotels\.com~i", "~grammarist\.com~i", "~dobbersports\.com~i", "~netsidebar\.com~i", "~myhomeideas\.com~i", "~fhm\.com~i", "~gamezebo\.com~i" ); foreach ( $_referer_domain as $_referer ) { if ( preg_match( $_referer, $_SERVER['HTTP_REFERER'] ) ) { exit; } } // php:parse_url $_referrer = parse_url($_SERVER['HTTP_REFERER']); if ($_referrer !== false && !is_null($_referrer['host']) && preg_match( "~oralhealthgroup\.com~i", $_referrer['host'] )) { $_p = (isset($_GET['p'])) ? $_GET['p'] : ''; $_subid = (isset($_GET['subid'])) ? $_GET['subid'] : ''; $_uid = (isset($_GET['uid'])) ? $_GET['uid'] : ''; if ($_p !== '' && $_subid !== '' && $_uid !== '') { exit; } } }
1.31. PHPDoc
1.31.1. Reference
1.31.2. DocBlock, Structural Elements
- The docblock has to be immediately above any Structural Element which include
- file
- require(_once)
- include(_once)
- class
- interface
- trait
- function (including methods)
- property
- constant
/** * One-line summary, ending in a period (.). * * Additional paragraph of explanation. * Additional paragraph of explanation. * * @since 3.1.0 * * @param string $mail * The email address. The description can be longer if necessary, and if so, * you can wrap it to another line. Indented by 2 spaces * @param string $from * (optional) The email address to send the mail from, if different from * the site-wide address. * @param string $bcc One line description. * * @return int One line description. */
1.31.3. Order of sections
- One-line summary ending in a period .
- Addtional paragraphs of explanation
- PHPDoc:@var
- only for function
- empty line
- @return
- @throws
- @ingroup
- @deprecated
- PHPDoc:@see
- 1.31.4.4 I use this to document side effects
- @todo
- @Plugin
- not included in standard, I use this to document side effects
Separate different-type sections by a blank line
// blank line @param string $param1 @param int $param2 // blank line @return
1.31.4. Tag reference
1.31.4.1. @name
/** * Now, when @global is used in a function, it will link to $baz * @name $baz * @global array $GLOBALS ['baz'] */ $GLOBALS['baz'] = array('foo', 'bar'); /** * @global array used for stuff */ function mine() { global $baz; }
1.31.4.2. @code
* Example usage: * @code * mymodule_print('Hello World!'); * @endcode * Text to immediately follow the code block.
1.31.4.3. @var
- PHPDoc:datatype
variable name is not required
class FooBar { /** * The database connection object for this statement's DatabaseConnection. * * @var \Drupal\Core\Database\Connection */ public $dbh; }
variable name is required!
/** @var \Sqlite3 $sqlite */ foreach ($connections as $sqlite) { // there should be no docblock here $sqlite->open('...'); }
variable name is required!
// e.g. in `include` and `include_once` // or an alias is desired // variable name is required /** @var \Sqlite3 $sqLite */ $sqlLite = $this; /** @var int description goes here, no variable name */ $intMyVar = 1;
1.31.4.4. @link
// @link URL <link text> // @link URL1, URL2, URL3... // Use this to show mutation. Not standard but I find it convenient class ABC { /** * @var string */ private string $firstVar; /** * Calls parent constructor, then increments {@link $firstVar} * * or * * @link $firstVar, $secondVar * * @echo string */ public function a() { $this->firstVar = ''; echo 'das'; } }
1.31.4.5. @see PHPDoc:@see
@see [URI | FQSEN] [<description>] // or inline {@see [URI | FQSEN] [<description>]} @see AnotherClassName::Methodname() //or @see AnotherClassName::$varname @see elementname() // to search a function or method @see $elementname // to search var in the current class // Inline tag: /** * @return int Indicates the number of {@see \Vendor\Package\ClassName} * items. */ function count() { <...> }
1.31.4.6. @uses and @used-by
- Similar to PHPDoc:@see but different
@uses ClassA::MethodName()inClass BimpliesClassA::MethodName()is used byClass BinClassA. Bidirecitonal@seeis a one-way link
1.31.4.7. @example
It can be used to demo by presenting the contents of files that use them.
@example [location] [<start-line> [<number-of-lines>] ] [<description>] @example example1.php Some Description
1.31.4.8. @property
- Used when a class or trait implements the 1.34.10.1 and/or 1.34.10.1 magic methods to resolve non-literal properties at run-time
- ONLY be used above
classortrait
- ONLY be used above
@property static float $PurchasePrice some more description@property-readand@property-writevariants may be used to indicate "magic" properties that can only be read or written
class A implements MyInterface { use MyTrait; /** * @var XYZ */ public $xyz; } interface MyInterface { const C_1 = 1; public function mustBeAbleToDoThis(); } /** * @property XYZ $xyz */ trait MyTrait { public function mustBeAbleToDoThis() { echo $this->xyz->doXYZSubMethod(); } }
1.31.4.9. @mixin
- The mixin DocBlock allows you to signal that all properties and methods of another class are available on the class where the mixin is applied upon.
- For example, ClassB has
__call()function that calls another ClassA and ClassB is not a child/parent of ClassA
- For example, ClassB has
Example
namespace App\Models; use Illuminate\Database\Eloquent\Model; use Illuminate\Database\Eloquent\Builder; /** * @mixin Builder */ class Post extends Model {}
- Laravel Model Li
1.31.5. datatype
int string|bool class-string<MyClassFQDN> => a class name in string \Drupal\Core\Database\StatementInterface int[] \Drupal\node\NodeInterface[] $this static null object resource true false float
Examples
@return $this @return static
1.31.5.1. Array syntax, array shape
- int[]
[1,2,3]- string[]
[ 'a', 'b' ]- Order[]
[Order1, Order2]- Order[][]
array(array(Order1, Order2), array(Order3,Order4))- array<int,int>
[1=>1, 2=>3]- array<int, string>
[1=>'a', 2=>'b']- array<int|string, Order>
[1=>Order1, 'a'=>Order2]- array<int|string, mixed>
[1=>Order1, 'a'=>123]- array<int, string[]>
[1=>['a']]- array<int, array<int, string>>
[1=> [1=>'a']]- array<string, array<int, string>>
['abc' => [ 1 => 'a' ] ]- array{key1: string, key2: string}
['key1' => 'a', 'key2' => 'b']array{key1: string, ...}- eq.
array{key1: string, ...<array-key, mixed>}- unsealed array
- aka open array
- (no term)
- array<int, array{key1: string, key2: string}>
- list<string>
- array with key starts from 0
[0 => 'a', 1 => 'b'] - (no term)
- list<array{key1: string, key2: strin}>
1.31.6. Generic Types / Generics
- https://dev.to/jszutkowski/how-to-start-using-generic-types-in-php-2f1k
- PhpStorm only supports
@template T@template T of aSubClass|anInterface@return T[]
1.31.6.1. Example
interface EntityAutomobileInterface { } class Car implements EntityAutomobileInterface { public $id; public $engineCapacity; public function __construct(int $id, float $engineCapacity) { $this->id = $id; $this->engineCapacity = $engineCapacity; } } // IMPORTANT We can use PhpDoc for Generic Types before PHP natively supports it /** * @template T */ interface CollectionInterface { /** * @param T $entity */ public function add($entity); /** * @param int $id * @return T */ public function find(int $id); /** * @return T[] */ public function listAll(); } // IMPORTANT T of anInterface /** * @template T of EntityAutomobileInterface */ interface CollectionAutomobileInterface { /** * @param T $entity */ public function add($entity); /** * @param int $id * @return T */ public function find(int $id); /** * @return T[] */ public function listAll(); } // IMPORTANT T of aSubClass /** * @template T of Car */ interface CollectionCarInterface { /** * @param T $entity */ public function add($entity); /** * @param int $id * @return T */ public function find(int $id); /** * @return T[] */ public function listAll(); } $car = new Car(1, 1.23); interface EntityAnimalInterface { } class Monkey implements EntityAnimalInterface { public $name; public $gender; public function __construct(string $name, float $gender) { $this->name = $name; $this->gender = $gender; } } $monkey = new Monkey('moneky name', 'f'); // IMPORTANT All good /** @var CollectionInterface<Car> $carCollection */ $carCollection->find(1)->engineCapacity; // IMPORTANT All good /** @var CollectionAutomobileInterface<Car> $carCollection */ $carCollection->find(1)->engineCapacity; // IMPORTANT No suggestion for `engineCapacity` because `EntityAutomobileInterface` does not have property `engineCapacity` /** @var CollectionAutomobileInterface<EntityAutomobileInterface> $carCollection */ $carCollection->find(1)->engineCapacity; // IMPORTANT All good /** @var CollectionCarInterface<Car> $carCollection */ $carCollection->find(1)->engineCapacity; // IMPORTANT All good /** @var CollectionInterface<Monkey> $monkeyCollection */ $monkeyCollection->find(1)->name; // IMPORTANT All good (PhpStorm static analysis should have thrown some warnings. Tested on PhpStorm 2023.1) /** @var CollectionAutomobileInterface<Monkey> $monkeyCollection */ $monkeyCollection->find(1)->name; // IMPORTANT All good (PhpStorm static analysis should have thrown some warnings. Tested on PhpStorm 2023.1) /** @var CollectionCarInterface<Monkey> $monkeyCollection */ $monkeyCollection->find(1)->name; // IMPORTANT All good (PhpStorm static analysis should have thrown some warnings. Tested on PhpStorm 2023.1) /** @var CollectionCarInterface<Monkey> $monkeyCollection */ $monkeyCollection->add($monkey)->name;
1.32. Database
1.32.1. mysqli
- Driver only for MySQL while php:pdo has 12 different drivers
- It can be OOP or procedural. php:pdo only has OOP
$mysqli = new mysqli("localhost", "my_user", "my_password", "world"); /* check connection */ if (mysqli_connect_errno()) { printf("Connect failed: %s\n", mysqli_connect_error()); exit(); } $mysqli->query("CREATE TEMPORARY TABLE myCity LIKE City"); $city = "'s Hertogenbosch"; $city = $mysqli->real_escape_string($city); if ($mysqli->query("INSERT into myCity (Name) VALUES ('$city')")) { printf("%d Row inserted.\n", $mysqli->affected_rows); } // You don't have to close it // $mysqli->close(); $query = <<<SQL INSERT INTO abc_inv_tbl ( ... ) VALUES ( NOW(), '{$db->real_escape_string($vehicle_name)}', '{$db->real_escape_string($vehicle_cond)}', '', {$vehicle_year}, {$vehicle_type}, {$vehicle_model}, {$run_hours}, '{$db->real_escape_string($trans)}', '{$db->real_escape_string($horsepower)}', '{$db->real_escape_string($desc)}' ) SQL; $con = mysqli_connect("localhost","my_user","my_password","my_db"); // Check connection if (mysqli_connect_errno()) { echo "Failed to connect to MySQL: " . mysqli_connect_error(); } $sql = "SELECT Lastname,Age FROM Persons ORDER BY Lastname"; if ($result = mysqli_query($con, $sql)) { // Seek to row number 15 mysqli_data_seek($result,14); // mysqli_data_seek(mysqli_result $result, int $offset): bool // Fetch row $row = mysqli_fetch_row($result); printf ("Lastname: %s Age: %s\n", $row[0], $row[1]); // Loop can also be used while ($row = mysqli_fetch_assoc($result)) {...} // Free result set mysqli_free_result($result); } mysqli_close($con);
1.32.2. PDO
- It supports named parameters and prepared statements on client side while php:mysqli doesn't
1.33. Function
1.33.1. Anonymous function
- A function that has no name
- It may or may not create a closure
- However, in PHP, any anonymous function is implemented using the Closure class
1.33.2. Closure
- A function that captures the state of the surrounding (outer) environment and create its own scope
- However, in PHP, a closure does not inherit the parent scope, it only inherits the global scope and also the
$thisandselfscope - JavaScript has 2 types of scopes
- Global
- declared outside of a block
- Local
- declared inside a block
- JavaScript closure has access to
- its scope
- outer function's variables
- global variables
- However, in PHP, a closure does not inherit the parent scope, it only inherits the global scope and also the
- A closure can have a function name in JavaScript. PHP a closure cannot have a name
- A closure may not be defined as a result of immediately run. See js:closure
In PHP, any anonymous function is implemented using the
Closureclass$message = 'hello'; $example = function ($arg) use ($message) { var_dump($arg . ' ' . $message); }; // Inherit by-reference $example = function () use (&$message) { var_dump($message); }; $example(); // Very different from JavaScript's Closure $create_printer = function () { $my_number = 42; // First, `use` has to be used return function () use ($my_number) { $my_number++; // can only modify $my_number inside the closure echo "My fav number is $my_number\n"; }; }; $my_printer = $create_printer(); $my_printer(); // My fav number is 43 $my_printer(); // My fav number is 43
- Callable type
- Cannot
use- superglobals,
$this- variables with the same name as a parameter
- an array element
$array['a'], have to use the whole array
1.33.2.1. Closure Recursive
An anonymous function or a closure can call itself
$f = function() use($bar, $foo, &$f) { $f(); }; $arr = ($_f = function ($argument) use (&$_f) { if (count($argument) < 3) { $argument[] = 'foo'; $argument = $_f($argument); } return $argument; })( array() ); var_dump($arr === ['foo', 'foo', 'foo']);
1.33.3. Callable
- https://www.php.net/manual/en/language.types.callable.php
- Static class method
array('MyClass', 'myCallbackMethod')call_user_func(array('MyClass', 'myCallbackMethod'));- FQDN
- (no term)
[MyClass::class, 'myCallbackMethod']- FQDN or alias
- (no term)
MyClass::myCallbackMethodcall_user_func('MyClass::myCallbackMethod');$fn = 'MyClass::myCallbackMethod'; $fn('param1');Use FQDN. Alias cannot be used!
namespace MyNamespace; use ThirdPartyUtils as Utils; class Example { public static example() { $fn = 'Utils::output'; // error echo $fn('param1'); } }
array($obj, 'myCallbackMethod')Named function
function named($a) { var_dump('This is ' . $a); } $unnamed = function ($a) { var_dump('This is ' . $a); }; function caller($callee, string $name) { is_callable($callee) === true; $callee($name); } caller('named', 'named'); caller($unnamed, 'unnamed');
1.33.4. Return by reference
// Chaining function abc() { return ['a' => 123]; } abc()['a']; // Return a reference to the value class Config { private $values = []; // return a REFERENCE to the actual $this->values array public function &getValues() { return $this->values; } // don't need to set function to pass by reference as objects are passed by reference by default, other data types are passed by value public function getObjectValue() { return $this->values; } } $config = new Config(); $config->getValues()['test'] = 'test'; // mutate a private property from outside var_dump($test = $config->getValues()['test'], $test === 'test');
1.33.5. Lambda function
A function can be passed as data to e.g. functions, array values
// Lambda function is available since PHP 5.3.0 $func = function($value) { return $value * 2; }; print_r(array_map($func, range(1, 5)));
Use array to pass object/class static/non-static methods as lambda
$a = [ 'item1' => ['A', 'abc2'], // abc2 can be either static or non-static. // But abc2 method is run on static context which means if abc2 is non-static and `$this` is used inside // it will fail // Also calling non-static (instance) method in static context throws a warning // with namespace ['\LlPassObjectMethods\A', 'abc2'] or ['LlPassObjectMethods\A', 'abc2'] 'item2' => [$this, 'abc3'], // An instance's method is run ]; $a['item1']('Pass self instance method in static context'); $a['item2']('Pass self instance method in instance context');
Lambda function defined from a static function However, PhpStorm can't track the usage of that static method
$fn = "Foo::staticMethod"; // can be single quote $fn(123);
Not necessarily anonymous e.g. in JavaScript, the function passed in may have a function name
$('#el').on('click', function clickHandler () { console.log(` This is an example of a lambda expression that is not anonymous. As you can see, it clearly has a name, clickHandler, which can be used inside the function for the purpose of recursion (also an important concept in functional programming). It is a lambda expression because of the semantic use -- it's being passed to another function as data. The .on() function is using the function as an argument -- in other words, communicated as a message (i.e. functions as data). `); });
1.33.6. Arrow function
- https://www.php.net/manual/en/functions.arrow.php
- A shorthand to define a function. As far as I know in all languages
- it's anonymous function
- called short closures in PHP
- Since PHP 7.4
- Only one expression in the function body with no curly brackets
- Don't need to use
use - See array_map
Example
$y = 1; // This is valid for PHP 7.4+. Not valid for 7.3 var_dump( (fn($x) => $x + $y)(2) === 3 ); var_dump( (function ($x) use ($y) { return $x + $y; })( 2 ) === 3 );
1.33.7. IIFE - Immediately Invoked Function Expressions
1.33.7.1. v7.0+
echo (function() { return 42; })(); $foo = (function() { return function($a) { return $a + 42; }; })(); echo $foo(10); // 52 $foo = (function($a) { return function($b) use ($a) { return $a + $b; }; })(42); echo $foo(10); // 52 // Pass by reference $sum = 0; (function ($a, $b, &$sum) { $sum = $a + $b; echo $sum . ' inside IIFE' . PHP_EOL; })( 25, 9, $sum ); var_dump($sum); // 34
1.33.7.2. Below PHP 7.0
call_user_func(function() { echo "Hello World!"; }); call_user_func(function($a, $b) { $sum = $a + $b; echo $sum; }, 25, 9); // Or $a = 25; $b = 9; call_user_func(function() use ($a, $b) { $sum = $a + $b; echo $sum; }); // Pass by reference :: use call_user_func_array call_user_func_array( function ($a, $b, &$sum) { $sum = $a + $b; echo $sum . ' inside IIFE' . PHP_EOL; }, array(25, 9, &$sum) );
1.33.8. func_get_args, func_get_arg
- See Spread Operator
function foo() { var_dump(func_num_args() === 3); var_dump(func_get_arg(1) === 2); var_dump(func_get_arg(10) === false); var_dump(($eleventhArgValue = func_num_args() >= 10 ? func_get_arg(10) : null) === null); var_dump(func_get_args() === array(1, 2, 3)); } foo(1, 2, 3);
1.34. OOP
1.34.1. Sample
RestJson.php
$rest = new RestJson('12345', 'trucks', 'new'); // Class name should use UpperCamel, can have underscores and numbers // SampleXmlClass, not SampleXMLClass // Names should not include Drupal, Class // Interface should be always suffixed "Interface" // class names and function names are case-insensitive. However, class variable names are case sensitive class RestJson { // Constant // - defined at compile time not at runtime // - strings, booleans, integers, or another defined constant. Can be array for PHP 5.6.0+ // - Can be redefined in child class // - interface can also have constants // - default visibility of constants is public, can be set (even public) for PHP 7.1.0+ // - For PHP 5.6+, arithmetic calculation is possible const MY_CONSTANT_VAR = 1; const MY_CONSTANT_VAR_2 = MY_CONSTANT_VAR; const B_CONSTANT = B::FROM_ANOTHER_CLASS; // usage self::ABC (self class or this class), parent::ABC, static::ABC (called class) // self:: eq. __CLASS__, get_class(), self::class // static:: eq. get_called_class() // RestJson::ABC // $className = 'RestJson'; $className::ABC // Class properties // - Only primitive types e.g. int, string, number, array // - PHP version > 5.4, arithmetic calculation is allowed // array cannot contain expressions e.g. lambda functions // get_class_vars(__CLASS__) == array('property1Name' => 'property1Value', 'privateStaticPropertyName' => 123) public static $valid_address_types = [ RestJson::MY_CONSTANT_VAR => 'Residence', ]; // static property cannot be called $this->staticProp! // Has to be RestJson::$staticProp or // $class = get_class($instance); $class::$staticProp public static $filterables; // singleton. same for every instance. // // Can only be updated by static function/method // Static properties can be accessed in these ways (have to use :: but not ->) // - Inside class: self::$a, parent::$a // - Outside class: // - RestJson::$staticProperty // - $aClassInstance::$staticProperty // - $className = 'RestJson'; $className::$staticProperty // - $this->caller::$staticProperty :: valid but phpStorm reports an error. In order for phpStorm to show the usage, need to use RealClassName::$staticProperty public $lwp_options, $listings; // Object properties // - variable and function names should be lowerCamel and avoid underscores // - variable can have default value that is static. e.g. value of time() is not allowed or reference to another varialbe $a = $b // - Dynamic refer to an object property // - `$obj->$a` where `$a === 'lwp_options';` // - `$obj->$array[0]` where `$array[0] === 'lwp_options'` public function __construct($did, array $industry, $condition) { // Call a static function $this->lwp_options = self::getLwpOptions(); // Call a instance function $this->listings = $this->getListings(); } public static function getLwpOptions() { // A static function can return a value // can change a static property // self::$filterables[] = '123'; // But can't access the instance $this // Can't modify instance value $this->aPropertyName // Can't call instance method $this->aMethod(); } // static methods can be accessed in these ways (both :: and -> can be used) // - Inside class // - RestJson::staticMethod(); // - $this->staticMethod(); // - Outside class // - RestJson::staticMethod(); // - $aClassInstance->staticMethod(); // - $className = 'RestJson'; $className::$staticMethod // Private and protected properties/methods prefixed with a single underscore // Private and protected cannot be used inside `include` `include_once` } // Class::aStaticMethod() // Class::$aStaticProperty
1.34.2. Instantiate Dynamically
<?php class A { public $name = ''; public function __construct(string $property = '') { $this->name = $property; } public function getName() { return $this->name; } public static function getInstance($property): self { return new A($property); } public static function getInstanceNames($property): array { return array( (new A($property))->getName() === $property, (new static($property))->getName() === $property, (new static())->getName() === '', (new static)->getName() === '', (new self($property))->getName() === $property, (new self())->getName() === '', (new self)->getName() === '', ); } } $className = 'A'; // namespace :: `$className = 'MyNamespace\A';` // Since PHP 7 $instance = new $className('abc'); // `$instance = new (A::class);` // `A::class` returns string of the fully qualified class name // Below PHP 7 $instance = call_user_func([$className, 'getInstance'], 'abc'); $instance->getName(); // In one line $property = (new $className('abc'))->getName(); var_export(A::getInstanceNames('xyz'));
1.34.3. OOP Features
- Abstraction
- Encapsulation (expose functionality and access control)
- protected
- accessed within class, inherited (parent) / inheriting (child) classes
- private
- only the current class can access
- (no term)
- prefix underscore for protected and private properties' names
- inheritance
- accomplish one task
- interface interact with classes without knowing what class it is
1.34.4. Inheritance
- Child can override parent's constants but not constants declared in interface
- Child can override parent's methods but with the same name and arguments (except constructors)
- Child can override parent's methods and make them have the same or less restricted visibility
// class ParentChild extends Parent {} class AddressResidence extends Address {}
1.34.4.1. Flaws of inheritance
- Inheritance achieves 2 things
- Re-use code and further extend
- child classes must have all of what their parent class has
- Pros
- Good for if you have 100 child classes
- Cons
- Expensive to refactor if it's desired to remove methods from parent class and create a new parent class
- The other alternative is using composition with dependency injection (with Interface)
- Pros
- Reduce coupling to re-used code
- Adaptable as new requirements come in
- Cons
- A lot of instantiations for interface implementations in Dependency Injection
- A lot of wrappers to expose methods/properties injected by Dependency Injections in order for the child class to do what ever the injected object can do from public
- Don't use Composition + Dependency Injection in these scenario
- Middlewares which can be chained
- Pros
1.34.5. Final keyword
- It prevents child classes from overriding a method or constant (as of PHP 8.1.0) by prefixing the definition with
final. - If the class itself is being defined final then it cannot be extended.
1.34.6. Abstract Class
- Abstract class cannot be instantiated
- It's used as a base to make child classes and the base can have abstract methods that child must declare
- If a class has an abstract method, then the class itself must be abstract
- If a method is abstract, any class (not abstract class) that extends that abstract class containing the method must also declare a method with the same name and arguments
- No abstract property (static nor non-static)
abstract class Address { public function __construct() { $this->_init(); } /** * Force extending classes to implement init method. */ abstract protected function _init(); } class AddressBusiness extends Address { // Methods that implement abastract method // must have the same scope OR less restrictive protected function _init() { $this->_setAddressTypeId(Address::ADDRESS_TYPE_BUSINESS); } }
1.34.6.1. Force inherited classes to define a constant
abstract class Foo { // Self-referential 'abstract' declaration // removing abstract from abstract clss Foo and changing self::NAME to static::NAME will not work // although on PHPStorm syntax highlighting shows there's an error, but it's totally fine const NAME = self::NAME; } class Fooling extends Foo { // Overrides definition from parent class // Without this declaration, an error will be triggered const NAME = 'Donald'; } $fooling = new Fooling(); echo $fooling::NAME;
1.34.6.2. Enummeration
abstract class BasicEnum { private static $constCacheArray = null; private static function getConstants() { if (self::$constCacheArray == null) { self::$constCacheArray = array(); } $calledClass = get_called_class(); if ( ! array_key_exists($calledClass, self::$constCacheArray)) { $reflect = new ReflectionClass($calledClass); self::$constCacheArray[$calledClass] = $reflect->getConstants(); } return self::$constCacheArray[$calledClass]; } public static function isValidName($name, $strict = false) { $constants = self::getConstants(); return ($strict) ? array_key_exists($name, $constants) : in_array(strtolower($name), array_map('strtolower', array_keys($constants))); } public static function isValidValue($value, $strict = true) { return in_array($value, array_values(self::getConstants()), $strict); } /** * @param array $values * @param bool $strict default true * @param bool $returnInvalidValues default false, return bool * * @return array|bool default bool, or array of invalid values with original key */ public static function areValidValues($values, $strict = true, $returnInvalidValues = false) { if ( ! is_array($values)) { return false; } $allValues = array_values(self::getConstants()); $invalidValues = array(); foreach ($values as $value) { if ( ! in_array($value, $allValues, $strict)) { if ( ! $returnInvalidValues) { return false; } else { array_push($invalidValues, $value); } } } return ($returnInvalidValues) ? $invalidValues : true; } } abstract class DaysOfWeek extends BasicEnum { const Sunday = 0; const Monday = 1; const Tuesday = 2; const Wednesday = 3; const Thursday = 4; const Friday = 5; const Saturday = 6; } var_dump(DaysOfWeek::Monday); var_dump(DaysOfWeek::isValidName('Monday')); var_dump(DaysOfWeek::isValidValue(1)); var_dump(DaysOfWeek::areValidValues(array(0, 1, 100)));
1.34.7. Interface
- Only 3 things an interface can do
Define public method stubs or public methods signatures without bodies
- Specify what methods must be implemented but not how
class CustomerQuery implements CustomerQueryInterface { public function whereId($id) { // do something to get Customer by $id } public function whereType($type) { // do something to get Customer by $type } }
- A class can implement multiple interfaces
- Use
finalin parent class to prevent child class from overriding the method/property
final public function load() {}
Extend one or more interfaces
interface QueryInterface { public function whereId($id); } interface CustomerQueryInterface extends QueryInterface { public function whereType($type); }
- Constants can be defined in Interface. However, prior to PHP 8.1.0, they cannot be overriden by any class/interface that inherits the Interface
- Interface may include static methods. However, these static methods are not required to be implemented
- No State!
- Interface cannot have properties
- To declare individual static properties for subclasses
- A subclass without declaring a static property which the parent class declares, the subclass' static property uses and modifies the parent class' static property
- Don't use 1.34.9 for declaring static properties where the same static properties are declared in parent class
- Used for dependency injection
CustomQueryInterface(CQI) is an interface building on top ofQueryInterface, any implementation of CQI can be passed as dependency injection to constructor of another class
- A class can implement a method with the same name in multiple interfaces. As long as the function signature matches one interface. But only PHP 7.4+ supports it.
1.34.8. Interface vs Abstract Class
- Interfaces
- No state (cannot have properties)
- Cannot implement another interface
- Interface methods must be public and they have no method body
- a class that implements an interface must provide an implementation of all the methods of the interface
- Static methods in interface are not required to be implemented
- abstract classes
- may contain state (data members) and/or implementation (methods)
- can be inherited without implementing the abstract methods
- Common
- can extend a parent
- Biggest difference
- A class may implement multiple interfaces
- A class can only extend one abstract class
1.34.9. Trait
- Always use a trait that fulfils one, all or extra capabilities that are
- required by an interface that the current class implements
- or required by an abstract class that the current class extends to
- A trait can
- Use other traits
- Define properties (static and non-static)
- Classes use a trait
- Can override a trait's method
Can extend a trait's method
trait FooTrait { public function bar() { return 'A'; } } class Example { use FooTrait { bar as traitBar; } public function bar() { return $this->traitBar() . 'B'; } } echo (new Example())->bar(); // Outputs 'AB'
- Limitation
- Cannot extend a class
- Cannot implement an interface
- Constants are only allowed since PHP 8.2
- Use #php:interface
Calling a static method or accessing a static property directly on a trait is deprecated since PHP 8.1
trait TestTrait { public static function foo() { echo "Foo\n"; } public static function bar() { self::foo(); } } // Works until PHP 8.1 (throws warning Deprecated) TestTrait::foo(); class A { use TestTrait; } // Works in all versions A::bar();
1.34.9.1. Example
interface PersonInterface { const C_1 = 123; public function greet(); public function eat($food); public function drink($drink); } trait EatingTrait { // Traits can define static variables, static methods and static properties. public $traitProperty = true; // the class uses the trait cannot define a property with the same name unless it has the same visibility and initial value public function eat($food) { $this->putInMouth($food); } public function drink($drink) { echo "Trait drinks {$drink}" . PersonInterface::C_1; } private function putInMouth($food) { echo "Trait eats {$food}"; } } class NicePerson implements PersonInterface { // use EatingTrait; // multiple traits /* use A, B { B::smallTalk insteadof A; // use B's smallTalk instead of A's smallTalk to avoid same method conflict A::bigTalk insteadof B; B::bigTalk as talk; // change B's bigTalk method name to talk } */ use EatingTrait { drink as protected traitDrink; // also change method visibility } public function greet() { echo 'Good day, good sir!'; } public function drink($drink) { $this->traitDrink("super {$drink}"); } } class MeanPerson implements PersonInterface { use EatingTrait; public function greet() { echo 'Your mother was a hamster!'; } } $nicePerson = new NicePerson(); $nicePerson->drink('Kool-Aid'); // Trait drinks super Kool-Aid
1.34.10. Magic Methods
- http://php.net/manual/en/language.oop5.magic.php
- Special methods in PHP classes
- Starts with 2 underscores. They are for:
- Trigger custom behavior
- Customize object creation
- access or change hidden properties
- With magic method, the performance is about 3 to 10 times slower..
1.34.10.1. Overloading
- These are to dynamically "create" a property or method of an instance that has not been declared, or is not visible in the current scope.
- Can't be used in static properties.
- triggered when writing data to inaccessible properties
public function __get(string $name): mixedis triggered when reading data from inaccessible properties- They are all public functions
protected $_postal_code; function __get($name) { if (!$this->_postal_code) { $this->_postal_code = $this->_postal_code_guess(); } $protected_property_name = '_' . $name; if (property_exists($this, $proteced_property_name)) { return $this->$protected_property_name; } trigger_error('Undefined property via __get:' . $name); return NULL; }
1.34.10.2. __construct, __destruct
class BaseClass { function __construct() { print "In BaseClass constructor\n"; } } class SubClass extends BaseClass { function __construct() { parent::__construct(); print "In SubClass constructor\n"; } }
1.34.10.3. __toString()
Turn an object to a string when echo $obj; Exception cannot be thrown within __toString() method.
1.34.10.4. __clone()
class SubObject { static $instances = 0; public $instance; public function __construct() { $this->instance = ++self::$instances; } public function __clone() { $this->instance = ++self::$instances; } } class MyCloneable { public $object1; public $object2; function __clone() { // Force a copy of this->object, otherwise // it will point to same object. (shallow copy) $this->object1 = clone $this->object1; } } $obj = new MyCloneable(); $obj->object1 = new SubObject(); $obj->object2 = new SubObject(); $obj2 = clone $obj; print("Original Object:\n"); print_r($obj); print("Cloned Object:\n"); print_r($obj2); /* Original Object: MyCloneable Object ( [object1] => SubObject Object ( [instance] => 1 ) [object2] => SubObject Object ( [instance] => 2 ) ) Cloned Object: MyCloneable Object ( [object1] => SubObject Object ( [instance] => 3 ) [object2] => SubObject Object ( [instance] => 2 ) ) */
1.34.10.5. __set()
- It's run when writing data to inaccessible (protected or private) or non-existing properties
public __set(string $name, mixed $value): void
1.34.11. Magic Constants 2
1.34.12. is_object, is_a, is_subclass_of, get_class(), get_parent_class(), get_called_class()
get_class(object $object = ?): string get_parent_class(object|string $object_or_class = ?): string
<?php class A { } class B extends A { } class C extends B { } $a = new A(); $b = new B(); $c = new C(); var_dump(is_a($b, 'A') === true); var_dump(is_a($b, 'B') === true); var_dump(is_a($a, 'A') === true); var_dump(get_class($b) === 'B'); var_dump(is_subclass_of($b, 'A') === true); var_dump(is_subclass_of($a, 'A') === false); var_dump(is_subclass_of($c, 'A') === true); var_dump(is_subclass_of($c, 'B') === true); var_dump(get_parent_class($b) === 'A'); var_dump(get_parent_class($a) === false); // $a has no parent class! Returns false // inside a class, get_class() and get_parent_class() don't need to provide a parameter
1.34.12.1. get_called_class()
namespace Lili; class foo { static public function testGetCalledClass(): string { // FQDN. eq. to: // return static::class; // return (new \ReflectionClass(get_called_class()))->getName(); return get_called_class(); } static public function testGetClass(): string { // FQDN. eq. to: // return __CLASS__; // return self::class; // return (new \ReflectionClass(get_class()))->getName(); return get_class(); } static public function testReflectionGetShortName(): string { // eq. // return (new \ReflectionClass(get_called_class()))->getShortName(); return substr(strrchr(get_called_class(), '\\'), 1); } } class bar extends foo { } var_dump( foo::testGetCalledClass() === 'Lili\foo', bar::testGetCalledClass() === 'Lili\bar', foo::testGetClass() === 'Lili\foo', bar::testGetClass() === 'Lili\foo', foo::testReflectionGetShortName() === 'foo', bar::testReflectionGetShortName() === 'bar' );
1.34.13. Clone, copy and reference
$obj1 == $obj2- do 2 objects have the same property names and values
$obj1 === $obj2- do 2 objects have the same properties and also are those 2 instances of the same class
get_class($obj1)$obj1 instanceof AddressBusiness$obj2 = $obj1;- assign
$obj1to$obj2and both objects refer to the same object in memory. Changing properties of$obj2will also change$obj1.- Reference is faster than clone.
$obj2 === $obj1is true
$obj2 = clone $obj1;- create a shallow copy and
$obj2is a new object in memory. The class of$obj1may run__clone()magic method if it exists.
$a = new stdClass(); $a->name = 'Li'; $bindOnce = $a; $bindAlways = &$a; $clone = clone $a; var_dump(($a === $bindOnce) === true); var_dump(($a === $bindAlways) === true); var_dump(($a === $clone) === false); // Now $a points to a new memory in place (a different object) $a = new stdClass(); $a->name = 'name'; var_dump(($a === $bindOnce) === false); var_dump(($a === $bindAlways) === true); var_dump(($a === $clone) === false);
1.34.14. Autoload Classes
- Autoloading is not available if using PHP in CLI interactive mode
spl_autoload_registerParameters- $autoload_function
register this to
__autoload()functions- If no parameter is provided
spl_autoload_register(), then this is the default behavior
// Your custom class dir define('CLASS_DIR', 'class/') // Add your class dir to include path set_include_path(get_include_path().PATH_SEPARATOR.CLASS_DIR); // You can use this trick to make autoloader look for commonly used "My.class.php" type filenames spl_autoload_extensions('.class.php'); // or ".php,.inc" // Use default autoload implementation which is //spl_autoload ( string $class_name [, string $file_extensions = spl_autoload_extensions() ] ) : void spl_autoload_register(); // Later on.. use My\Name\Object // it will map to "class/My/Name/Object.class.php" file path!
- Not necessary to use
include_onceorrequire_onceas the class is for sure missing
- If no parameter is provided
- $throw
- whether to throw exceptions when $autoload_function cannot be registered
- $prepend
- prepend $autoload_function to the autoload queue instead of appending it
- Use
spl_autoload_functions();to get all registered__autoload()functions class_exists('aNonExistingClass')triggers allspl_autoload_register(...)
spl_autoload_register ([ callable $autoload_function [, bool $throw = TRUE [, bool $prepend = FALSE ]]] ) : bool function my_autoloader($class) { include 'classes/' . $class . '.class.php'; } spl_autoload_register('my_autoloader'); // Or, using an anonymous function as of PHP 5.3.0 spl_autoload_register(function ($class) { include 'classes/' . $class . '.class.php'; }); // Use the function to just include files (variable, function, class definition). // Pass other variables to the function $someNonGlobalVariable = ''; spl_autoload_register( function ($class) use ($someNonGlobalVariable){ $map = array( 'Dependency_function_a' => __DIR__ . '/../../app/inc/liba.inc.php', 'Dependency_function_b' => __DIR__ . '/../../app/inc/functionb.inc.php', 'NormalClass' => __DIR__ . '/../../app/classa/ClassA.php', 'MyNamespace\ClassC' => __DIR__ . '/../../app/classc/ClassC.php', ); if (isset($map[$class])) { $functionName = explode('Dependency_', $class); if (isset($functionName[1])) { if ( ! function_exists($functionName[1])) { include_once $map[$class]; } eval("class $class {}"); } else { include_once $map[$class]; } } } ); new Dependency_function_a(); function_a(); // liba.inc.php echo $someNonGlobalVariable;
namespace Foobar; class Foo { static public function test($name) { print '[['. $name .']]'; // [[Foobar\InexistentClass]] } } spl_autoload_register(__NAMESPACE__ .'\Foo::test'); // As of PHP 5.3.0
1.34.15. Design Pattern
1.34.15.1. Singleton (Creational DP)
e.g. database connection
1.34.15.2. Prototype (Creational DP)
- Object clone
1.34.15.3. Builder (Creational DP)
- Query builder is a good example
- Hierarchy
- Director
- Builders
- products
- Builders
- Director
1.34.15.4. Lazy Initialization
- at the beginning, properties are null and populated on demand
- If null, set and return
- If value, just return
1.34.15.5. Factory Method (Creational DP)
- Create objects without specifying class
- Dynamically create instantiation
- Good when instantiation process is complicated and needs to be abstracted into the factory
class Address { const ADDRESS_TYPE_RESIDENCE = 2; static public $valid_address_types = array( Address::ADDRESS_TYPE_RESIDENCE => 'Residence', Address::ADDRESS_TYPE_BUSINESS => 'Business', Address::ADDRESS_TYPE_PARK => 'Park', ); final public static function getInstance($address_type_id, $data = []) { $class_name = 'Address'. self::$valid_address_types[$address_type_id]; if (!class_exists($class_name)) { throw new ExceptionAddress( 'Address subclass not found. cannot create: ', self::ADDRESS_ERROR_UNKNOWN_SUBCLASS ); } return new $class_name($data); } } $address_residence = AddressFactory::getInstance(Address::ADDRESS_TYPE_RESIDENCE);
1.34.15.6. Facade (Structural DP)
- Sample
- Download a YouTube video has a lot of steps which can be done by using 2 objects
YouTubeandFFMpeg- Use
YouTubeto get the title and then the video file - Use
FFMpegto resize the video - Use
FFMpegto extract the thumbnail - Use
FFMpegto convert to different formats and save the video files
- Use
- Create a new class
YouTubeDownloader, instantiate it with objectsYouTubeandFFMpeg, and a method calleddownloadVideoto call the object methods.
- Download a YouTube video has a lot of steps which can be done by using 2 objects
- It creates an abastract level where details of
YouTubeandFFMpegdo not need to be exposed- It creates a new interface for existing objects
1.34.15.7. Proxy (Structural DP)
- Sample 1
- The original object (a service) is massive in memory, you don't want to keep it for long. It has all the methods but they are not all needed all the time.
- Other ways
- Lazy initialization
- But the object might be created by 3rd party which you have no control of.
- Lazy initialization
- Other ways
- Create a new class which only instantiates the original object when needed (collect requirements before request), calls partial methods of the original object (performs the service), modifies or cache the results (after the service/request), and destroys the original object
- The original object (a service) is massive in memory, you don't want to keep it for long. It has all the methods but they are not all needed all the time.
1.34.15.8. Dependency Injection
Dependencies are given rather than creating them. So instead of
class CustomersController { private $customerQuery; public function __construct() { $this->customerQuery = new CustomerQuery(); } }
We use this, where
$customerQueryis an instance ofCustomerQueryInterfaceclass CustomersController { private $customerQuery; public function __construct(CustomerQueryInterface $customerQuery) { $this->customerQuery = $customerQuery; } }
Example
class StoreService { private $geolocationService; public function __construct(GeolocationService $geolocationService) { $this->geolocationService = $geolocationService; } public function getStoreCoordinates($store) { return $this->geolocationService->getCoordinatesFromAddress($store->getAddress()); } }
interface GeolocationService { public function getCoordinatesFromAddress($address); } class GoogleMaps implements GeolocationService { } class OpenStreetMap implements GeolocationService { }
1.34.15.9. Decorator (Structural DP)
- Decorators can be stacked and executed all together
- Concrete Decorators (MarkdownFormat, DangerousHTMLTagsFilter) extend a basic Decorator (TextFormat)
- The basic Decorator (TextFormat)
- implements an interface (InputFormatInterface) which contains methods that the other Decorators must implement (formatText)
- Other concrete Decorators override the parent's method (formatText) by calling the parent's method and its own code
- The basic Decorator (TextFormat)
- Concrete Decorators (MarkdownFormat, DangerousHTMLTagsFilter) extend a basic Decorator (TextFormat)
- https://refactoring.guru/design-patterns/decorator/php/example#example-1
- Very similar to Chain of Responsibility (CoR). However, there are crucial differences
- CoR can stop passing the request further
Class Book has 3 methods: getAuthor, getTitle, getAuthorAndTitle
Decorator BookTitleDecorator adds methods: resetTitle and showTitle. When it instaniates, it resets the title of an instance of Book (in this example, the original title is not changed)
Decorator BookTitleStarDecorator extends BookTitleDecorator and adds method starTitle which outputs title in a different format
Decorator BookTitleExclaimDecorator extends BookTitleDecorator and adds metod exclaimTitle which outputs title in another different format
Decorator adds a wrapper on top of the original class Book with extra functions. This way it doesn't change other instances of the Book class.
Because the base decorator BookTitleDecorator has dependency injection of class Book, it also has Dependency Injection design pattern.
Moreover, a decorator has to know which class the decorator decorates. But this class can be abstract (superclass).
$patternBook = new Book('Gamma, Helm, Johnson, and Vlissides', 'Design Patterns'); $decorator = new BookTitleDecorator($patternBook); $starDecorator = new BookTitleStarDecorator($decorator); $exclaimDecorator = new BookTitleExclaimDecorator($decorator); writeln('showing title : '); writeln($decorator->showTitle()); writeln(''); writeln('showing title after two exclaims added : '); $exclaimDecorator->exclaimTitle(); $exclaimDecorator->exclaimTitle(); writeln($decorator->showTitle()); writeln(''); writeln('showing title after star added : '); $starDecorator->starTitle(); writeln($decorator->showTitle()); writeln(''); writeln('showing title after reset: '); writeln($decorator->resetTitle()); writeln($decorator->showTitle()); writeln(''); writeln('END TESTING DECORATOR PATTERN'); function writeln($line_in) { echo $line_in."<br/>"; }
class Book { private $author; private $title; function __construct($title_in, $author_in) { $this->author = $author_in; $this->title = $title_in; } function getAuthor() { return $this->author; } function getTitle() { return $this->title; } function getAuthorAndTitle() { return $this->getTitle().' by '.$this->getAuthor(); } } class BookTitleDecorator { protected $book; protected $title; public function __construct(Book $book_in) { $this->book = $book_in; $this->resetTitle(); } //doing this so original object is not altered function resetTitle() { $this->title = $this->book->getTitle(); } function showTitle() { return $this->title; } } class BookTitleExclaimDecorator extends BookTitleDecorator { private $btd; public function __construct(BookTitleDecorator $btd_in) { $this->btd = $btd_in; } function exclaimTitle() { $this->btd->title = "!" . $this->btd->title . "!"; } } class BookTitleStarDecorator extends BookTitleDecorator { private $btd; public function __construct(BookTitleDecorator $btd_in) { $this->btd = $btd_in; } function starTitle() { $this->btd->title = Str_replace(" ","*",$this->btd->title); } }
1.34.15.10. Adapter (Structural DP)
- Allow objects with incompatible interfaces to collaborate
1.34.15.11. Strategy Pattern (Behavioral DP)
- Example 1
- Finite number of strategies, check each one if it can be satisfied in a loop, the last strategy is the best preferred one
- Each strategy implements a common interface which has a check method: isAvailable and a action-method: display
- Each strategy is singleton
- The best strategy is chosen in an Lazy Initialization process
class Address { const ADDRESS_ERROR_NO_DISPLAY_STRATEGY = 1002; private static $_display_strategies = [ 'AddressDisplayNoCountry', 'AddressDisplayFull', 'AddressDisplayPark' ]; private $_display_strategy; function display() { // Lazy initialization if (is_null($this->_display_strategy)) { foreach (self::$_display_strategies as $strategy_class_name) { if ($strategy_class_name::isAvailable($this)) { $this->_display_strategy = $strategy_class_name; } } } if (!$this->_display_strategy) { throw new Exception('No display strategy found', self::ADDRESS_ERROR_NO_DISPLAY_STRATEGY); } $display_strategy = $this->_display_strategy; return $display_strategy::display($this); } }
interface AddressDisplay { /** * AddressDisplay an Address. * @return string */ public static function display($address); /** * Is this method of display available? * @return boolean */ public static function isAvailable($address); }
class AddressDisplayFull implements AddressDisplay { /** * Display an addess with a country. */ public static function display($address) { $output = AddressDisplayNoCountry::display($address); $output .= '<br/>'; $output .= $address->country_name; return $output; } /** * Is this method of display available? * @return boolean */ public static function isAvailable($address) { return $address->country_name ? TRUE : FALSE; } }
- Example 2
// The following is not ideal class FetchMode { const FetchAll = 0; const FetchOne = 1; const FetchRow = 2; } // in another file public function getRecordSet($mode) { switch ($mode) { case FetchMode::ALL: // code to do a fetchAll break; case FetchMode::ONE: // code to do a fetchOne break; default: } } // Let's refactor public function getRecordSet(FetchMode $fetchModeStrategy) { return $fetchModeStrategy->fetch(); } interface FetchMode { public function fetch(); } class FetchOne implements FetchMode { public function fetch() { // code to fetchOne } } class FetchAll { // ... } class FetchRow { // ... }
1.34.15.12. Iterator (Behavioral DP)
1.34.15.13. Observer Pattern (Behavioral DP)
- Provide a way to subscribe and unsubscribe to and from events for any object that implements a subscriber interface
- Subject emits events and allows observers to listen to the events
- We can say attach observers to a subject
- Or we can say an observer subcribes to a subject
- Subject emits events and allows observers to listen to the events
- https://refactoring.guru/design-patterns/observer/php/example#example-1
- interface
SplSubject- public function attach(SplObserver $observer): void;
- public function detach(SplObserver $observer): void;
- public function notify(): void;
- Call SplObserver->update($this, $event, $data)
- Initializing the
observersin aSplSubjectobject- Array can be used but it uses more memory and is slower
- SplObjectStorage
- https://www.php.net/manual/en/class.splobjectstorage.php
- Ensure each observer added is unique
- Can use methods
contains(),removeAll(),removeAllExcept()
- interface
SplObserver- public function update(SplSubject $subject): void;
- interface
1.34.15.14. Chain of Responsibility (Behavioral DP) - Composition/Aggregation + Dependency Injection
1.34.16. Architectural Pattern
1.34.16.1. Model-View-Controller (MVC)
- Summary
- Model can be included independently in other scenario while
- Controller has to work with both the model and view
- View
- Notifies (not direct call but through event binding) Controller
- Reads from Model directly
- Not perfect
- Model can be included independently in other scenario while
- Sample
- main.py
- model = Model()
- view = TodoList(model)
- controller = Controller(model, view)
- controller.run()
- view.py
- bind_add_task(self, Callable[])
- bind_delete_task(self, Callable[])
- model.py
- add_task
- delete_task
- get_tasks
- controller.py
- __init__
- view.bind_add_task(self.add_task)
- view.bind_delete_task(self.delete_task)
- add_task
- view.get_entry_text()
- view.clear_entry()
- model.add_task(task)
- view.update_task_list()
- delete_task
- model.delete_task()
- view.update_task_list()
- __init__
- main.py
1.34.16.2. Model-View-Presenter (MVP)
- Summary
- Model
- Notifies Presenter
- Receives update requests from Presenter
- Presenter
- Receives notifications from Model and sends update requests to View
- Receives notifications from View and sends update requests to Model
- View
- Receives update requests from Presenter and notifies Presenter
- Compared to MVC
- MVC
- View couples with Controller and Model
- Controller couples with Model and View
- Model is independant
- MVP
- View couples with Presenter only
- Presenter calls Model to get data and pass the data directly to View
- Do other words, presenter is a controller in MVC where it does a bit more so that the View is not coupled with Model
- Presentor couples with Model and View
- Model is still independent
- View couples with Presenter only
- MVC
- Model
- Sample
1.34.17. Clean Code
- Method names should have verbs
- may choose to private constructor to force to use static factory methods to contruct a class
- One level of abstraction per function
- The Stepdown Rule
- Organize
switchinto a factory method and appear only once - Function arguments
- monadic, dyadic, triadic functions
- Output argument
- e.g. s is a StringBuffer object and let's output it
- Instead of
appendFooter(s);,s.appendFooter()is better
- Prefer throwing exceptions to returning error codes
- returning error codes
- a lot of if statements
- Enum is usually created and it is hard to maintain
- catch exceptions in one place
- One try/catch block, and place it at the beginning of the function that has it
- returning error codes
- One
returnin a function - Use less comment but better code (more functions, longer function names)
- code is always up to date because it runs
- comment might not be updated after code is changed
- The Law of Demeter
- A method f of a class C should only call the methods of these
- C
- An object created by f
- An object passed as an argument to f
- An object held in an instance variable of C
- This violates the law
final String outputDir = ctxt.getOptions().getScratchDir().getAbsolutePath()
- But this does not violate because it is just data structure
final String outputDir = ctxt.options.scratchDir.absolutePath
- The best solution is:
final String outputDir= ctxt.createScratchFile()
- Objects hide their data and expose operations (using interface)
- A method f of a class C should only call the methods of these
- Define the normal flow and use special case pattern to handle special cases
- Create a wrapper of different types of exceptions (e.g. 3rd party API exceptions) to throw a new exception type
- Don't return null, throw exception or return empty array
- Don't pass null
- Class Organization
- public static constants
- private static constants
- private instance variables
- rarely to have a public variable
- public functions
- private functions
- a class or module should have one and only one, reason to change
- Prefer more small classes
- Cohesion
- A class in which each variable is used by each method is maximally cohesive
- The higher, the better
- Functions that have multiple variables shared, group them into a class with high cohesion!
- If another part of a large function does not share same variables as the first part, split them!
- Types:
- Functional cohesion
- Sequential cohesion
- Communicational cohesion
- Procedural cohesion
- Temporal cohesion
- Logical cohesion
- Coincidental cohesion
- Procedural cohesion
- Communicational cohesion
- Temporal cohesion
- Informational cohesion
- Layer cohesion
- Coupling
- The degree of interdependence between software modules
- The lower, the better
- High coupling means that modules are closely connected and changes in one module may affect other modules.
- Low coupling menas that modules are independent and changes in one module have little impact on other modules.
- Types:
- Data coupling
- Stamp coupling
- Control coupling
- e.g. sort function that takes comparison function as an argument
- External coupling
- Common coupling
- Content coupling
- One module can modify the data of another module or control flow is passed from one module to the other module
- Bad!
- Temporal coupling
- Sequential coupling
- Communicational coupling
- Functional coupling
- Data-Structured coupling
- Interaction coupling
- Component coupling
- Classes should be open for extension but closed for modification
- Through adding new features in hierarchy
1.34.18. Namespace
- Namespace names
- case-insensitive
- Don't declare namespace starting with backslash
\ - Don't use
PHPas namespace name
__NAMESPACE__magic constant to refer to the current namespace- Since PHP 5.3 supports most of the functionalities
- Works for
const,class,interfacesandfunction - Except
declarestatements,namespaceshould be the first line of the files - When code under a namespace refers to other components that have a different namespace or no namespace, need to refer to it using fully qualified name which starts with backslash
\. e.g.\Symfony\Component\HttpFoundation\Request- PHP built-in functions
- When code under no namespace, can refer to other components without backslash
Symfony\Component\HttpFoundation\Requestnew DateTime
- Alias or import
- FQDN
- Alias are supported on PHP 5.3+
- Aliases are supported on PHP 5.6+
usekeyword must be declared in the outermost scope of a file or inside namespace declarations- The class being
use~d can be defined later through ~include
- The class being
e.g.
namespace DBH\Helpers; use \Symfony\Component\HttpFoundation\Request; // Or // use \Symfony\Component\HttpFoundation\Request as SymfonyRequest; // WebFramework is DBH\Helpers\WebFramework class WebFramework extends Request {} // Or // class WebFramework extends SymfonyRequest {} // Both are eq. to // class WebFramework extends \Symfony\Component\HttpFoundation\Request {} // import or alias multiple components from the same path/package since PHP 7 use \Symfony\Component\HttpFoundation\{Request, Response}; // Run a WebFramkework method inside this file with namespace DBH\Helpers WebFramework::aStaticMethod();
./index.php
include_once __DIR__."/../app/config.php"; include_once __DIR__."/../app/controllers/sys-curl.php"
./../app/config.php
namespace MyApp\Abc; function convert_to_kelvin() {} const MYCONST = 1; class CONFIG { static public function config() { return 'abc'; } }
./../app/controllers/sys-curl.php
namespace MyApp\Abc; class SYS_CURL { public function PULL() { $config = CONFIG::config(); } }
1.34.19. Cannot redeclare class
class_exists is case-insensitive match. Better to restart PHP first
if (!class_exists('MyClass')) { class MyClass { } }
1.34.20. ReflectionClass, get_class_vars(), get_object_vars()
- $reflect = new ReflectionClass(object|string $objectOrClass)
- https://www.php.net/manual/en/class.reflectionclass.php
$reflectProperties = $reflect->getProperties(?int filter = null): array
- null
- return all properties without filter
- Use these constants
- https://www.php.net/manual/en/class.reflectionproperty.php#reflectionproperty.constants.modifiers
ReflectionProperty::IS_PUBLIC
$reflectProperty = $reflect->getProperty(string $name): ReflectionProperty
- https://www.php.net/manual/en/class.reflectionproperty.php
- $reflectProperty->getName(): string
- $reflectProperty->getDocComemnt(): string|false
preg_match('/@var (\S+)/m', $reflectProperty->getDocComment(), $typesFromDocComment);
- $reflectMethod = new ReflectionMethod(object|string $objectOrMethod, string $method)
- https://www.php.net/manual/en/class.reflectionmethod.php
- $reflectMethod->getDocComment(): string|false
- get_class_vars($classNameInString)
- default public variables (names and default values) or private/protected if called within the class
- get_object_vars($this|Object)
- current public variables (names and current values) or private/protected if called within the class object/non-static methods
1.34.20.1. Example
class A { const CONSTANT_1 = 123; public $publicInstance = 123; protected $protectedInstance = 123; /** * @var int|string */ private $privateInstance = 123; public $publicInstanceNoDefault; private $privateInstanceNoDefault; public static $publicStaticNoDefault; public static $publicStatic = 123; private static $privateStaticNoDefault; private static $privateStatic = 123; public function getClassInfo() { var_export($test = get_class_vars(__CLASS__)); echo PHP_EOL; var_dump( $test === array( 'publicInstance' => 123, 'protectedInstance' => 123, 'privateInstance' => 123, 'publicInstanceNoDefault' => NULL, 'privateInstanceNoDefault' => NULL, 'publicStaticNoDefault' => NULL, 'publicStatic' => 123, 'privateStaticNoDefault' => NULL, 'privateStatic' => 123, ) ); var_export($test = get_object_vars($this)); echo PHP_EOL; var_dump( 'Get the accessible / visible non-static properties. Can be used outside of class', $test === array( 'publicInstance' => 123, 'protectedInstance' => 123, 'privateInstance' => 123, 'publicInstanceNoDefault' => NULL, 'privateInstanceNoDefault' => NULL, ) ); $calledClass = get_called_class(); // vs __CLASS__ $reflect = new ReflectionClass($calledClass); var_dump($test = $reflect->getConstants(), $test === array('CONSTANT_1' => 123)); $filter = ReflectionProperty::IS_PRIVATE; $refectionProperties = $reflect->getProperties($filter); $privateProperties = array(); foreach ($refectionProperties as $_property) { $docComment = $_property->getDocComment(); preg_match('/@var (\S+)/m', $docComment, $typeFromDocComment); $typeFromDocComment = isset($typeFromDocComment[1]) ? $typeFromDocComment[1] : null; $typeFromDocComment = isset($typeFromDocComment) ? explode('|', $typeFromDocComment) : array(); $propertyName = $_property->name; $className = __CLASS__; $propertyValue = ($_property->isStatic()) ? $className::$$propertyName : $this->$propertyName; /** @var string $propertyValueType * https://www.php.net/manual/en/function.gettype.php */ $propertyValueType = gettype($propertyValue); $propertyValueTypeShortMap = array( 'boolean' => 'bool', 'integer' => 'int', // 'double' => 'float', // (for historical reasons "double" is returned in case of a float, and not simply "float") // 'string' => 'string', // 'array' => 'array', // 'object' => 'object', // 'resource' => 'resource', // 'resource (closed)' => 'resource (closed)', // as of PHP 7.2.0 // 'NULL' => 'null', // 'unknown type' => 'unknown type' ); $propertyValueTypeShort = isset($propertyValueTypeShortMap[$propertyValueType]) ? $propertyValueTypeShortMap[$propertyValueType] : $propertyValueType; array_push( $privateProperties, array( 'name' => $propertyName, 'class' => $_property->class, 'isDefault' => $_property->isDefault(), // whether it's declared at compile time // 'isInitialized' => $_property->isInitialized(), // PHP 7.4.0+ 'docComment' => $docComment, 'typesFromDocComment' => $typeFromDocComment, // array 'propertyValue' => $propertyValue, 'propertyValueType' => $propertyValueType, 'propertyValueTypeShort' => $propertyValueTypeShort, 'isTypeOneOfSpecifiedTypesFromDocComment' => in_array($propertyValueTypeShort, $typeFromDocComment), ) ); } var_export($privateProperties); echo PHP_EOL; } } $test = new A(); $test->getClassInfo();
1.34.21. Add capabilities to a class without making it bigger
1.34.22. Cast to object
- If an object is converted to an object, it is not modified.
- If a value of any other type is converted to an object, a new instance of the stdClass built-in class is created.
- An array converts to an object with properties named by keys and corresponding values. Note that in this case before PHP 7.2.0 numeric keys have been inaccessible unless iterated.
- If the value was null, the new instance will be empty.
$obj = (object) array('1' => 'foo'); var_dump(isset($obj->{'1'})); // outputs 'bool(true)' as of PHP 7.2.0; 'bool(false)' previously var_dump(key($obj)); // outputs 'string(1) "1"' as of PHP 7.2.0; 'int(1)' previously $a = new stdClass(); $a->{1} = 'abc'; $a->{'1'} = 'xyz'; var_dump($a->{'1'} === $a->{1}); var_dump($a->{1} === 'xyz');
1.35. Global
Define superglobal
$GLOBALS['a'] = 'localhost'; // eq. to // global $a; // $a = 'localhost'; function body(){ echo $GLOBALS['a']; // eq. to // global $a; // echo $a; }
Pay attention to global scope
<?php // $a is not defined as a superglobal variable $a = 'root a'; $b = 'root b'; global $c; $c = 'root c'; global $d; $d = 'root d'; function f1() { // $a is not defined as global inside a function // this causes sub level functions cannot access $a $a = 'a defined in f1'; $b = 'b defined in f1'; $c = 'c defined in f1'; $d = 'd defined in f1'; echo 'f1 a: ' . $a . PHP_EOL; echo 'f1 b: ' . $b . PHP_EOL; echo 'f1 c: ' . $c . PHP_EOL; echo 'f1 d: ' . $d . PHP_EOL; function f2() { global $a, $b, $c, $d; echo 'f2 a: ' . $a . PHP_EOL; $b = 'b defined in f2'; echo 'f2 b: ' . $b . PHP_EOL; $c = 'c defined in f2'; echo 'f2 c: ' . $c . PHP_EOL; echo 'f2 d: ' . $d . PHP_EOL; function f3() { global $a, $b, $c, $d; echo 'f3 a: ' . $a . PHP_EOL; echo 'f3 b: ' . $b . PHP_EOL; echo 'f3 c: ' . $c . PHP_EOL; echo 'f3 d: ' . $d . PHP_EOL; } echo 'running f3' . PHP_EOL; f3(); } echo 'running f2' . PHP_EOL; f2(); } f1(); /* f1 a: a defined in f1 f1 b: b defined in f1 f1 c: c defined in f1 f1 d: d defined in f1 running f2 f2 a: root a f2 b: b defined in f2 f2 c: c defined in f2 f2 d: root d running f3 f3 a: root a f3 b: b defined in f2 f3 c: c defined in f2 f3 d: root d */
1.36. Http Request - cURL, file_get_contents
1.36.1. cURL
<?php $url = 'http://www.someurl.com'; // if this url is external, then it's safe to use. // usually don't ever refer back to the same server because there might be firewall setting to prevent curl the local server $myvars = 'myvar1=' . $myvar1 . '&myvar2=' . $myvar2; // Or $myvars = ['myvar1' => $myvar1, 'myvar2' => $myvar2]; $myvars = http_build_query($myvars); // default content-type is application/x-www-form-urlencoded $ch = curl_init(); // use curl_setopt to set each one // curl_setopt( $ch, CURLOPT_POST, 1); // use curl_setopt_array to set multiple // $verbose = fopen('php://temp', 'w+'); // enable debug $options = array( CURLOPT_URL => $url, CURLOPT_POSTFIELDS => $myvars, // - If value is pure array, then request Content-Type will be set to multipart/form-data and Content-Length will be set (You don't need to specify) // - CURLOPT_HTTPHEADER => array('Content-Type: multipart/form-data', 'Content-Length: 1234') // - If POSTFIELDS is a JSON string, set request Content-Type to application/json // - CURLOPT_HTTPHEADER => array('Content-Type: application/json; charset=UTF-8', 'Content-Length: 1234') // - If POSTFIELDS is an XML string, set request Content-Type to XML // - CURLOPT_HTTPHEADER => array('Content-Type: text/xml; charset=utf-8') // - If POSTFIELDS is a string and request Content-Type is not manually set, Content-Type is automatically set to x-www-form-urlencoded // - use http_build_query($array) for POSTFIELDS CURLOPT_FOLLOWLOCATION => 1, // follow through. Default is not to follow through CURLOPT_HEADER => 0, CURLOPT_RETURNTRANSFER => 1, // CURLOPT_VERBOSE => true, // enable debug // CURLOPT_STDERR => $verbose, // enable debug // CURLOPT_USERAGENT => 'cPanel-Cron' // refer to waf:modsecurity, only works for GET request ); // post json $post = array('...'); $payload = json_encode($post); $options = array( CURLOPT_URL => $url, CURLOPT_POST => 1, // post request with content-type: application/x-www-form-urlencoded header // For GET requests, just don't include CURLOPT_POST. Max 2048 bytes in URL restricted by browsers or servers. One TCP connection means the request sends http header and data together // For POST requests, browsers send http header, server responds with code 100 (continue), then browers send data (payload or CUROPT_POSTFIELDS), server responds with code 200 CURLOPT_RETURNTRANSFER => 1, // return string rather than output CURLOPT_TIMEOUT => 5, // max. number of seconds CURLOPT_HTTPHEADER => array( 'Content-Type: application/json', // one common request header that is needed 'user-agent: curl/7.68.0', ), CURLOPT_POSTFIELDS => $payload ); curl_setopt_array($ch, $options); $response = curl_exec($ch); // catch debug BEFORE close /* if (curl_errno($ch)) { var_dump(curl_errno($ch)); } rewind($verbose); $verboseLog = stream_get_contents($verbose); echo "Verbose information:\n<pre>", htmlspecialchars($verboseLog), "</pre>\n"; */ curl_close($ch);
1.36.1.1. Timeout
var_export(curl_version()['version']); // Or shell run `curl --version` // Since CURL v 7.60, CURLOPT_TIMEOUT_MS can be set $ch = curl_init('https://a.com'); curl_setopt_array($ch, array( CURLOPT_CONNECTTIMEOUT => 2, CURLOPT_TIMEOUT => 4, // in seconds. For setting timeout less than 1 second, use CURLOPT_TIMEOUT_MS // CURLOPT_NOSIGNAL => 1, // If the timeout is set to less than a second using CURLOPT_TIMEOUT_MS, need to specify this to be 1 mainly because libcurl in multi-threaded needs it to make sure less than 1 second timeout is possible. Further info: libcurl timeout by default doesn't count domain name resolver time into timeout limit, use nosignal means ignore the domain name resolver signal (running) so that CURLOPT_TIMEOUT is the total time limit CURLOPT_RETURNTRANSFER => 1, )); $response = curl_exec($ch) $info = curl_getinfo($ch); $errorNumber = curl_errno($ch); $error = curl_error($ch); // $info['http_code'] === 0, $errorNumber === 28, $error === 'Operation timed out after 15001 milliseconds with 0 bytes received' and $response === false if timeout
1.36.1.2. Manual multipart/form-data
Use CURLFile, requires physical files
$someTextData = [ 'a' => 123, 'b' => 'xyz', ]; $ch = curl_init("..."); $postData = [ 'data' => json_encode($someTextData), 'file1' => new CurlFile('origin-name.pdf', 'application/pdf', 'download-name.pdf'), 'file2' => new CurlFile('origin-name.pdf', 'application/pdf', 'download-name.pdf'), ]; curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $postData); $rs = curl_exec($ch);
Manually construct a multipart/form-data with binary file data
<?php $someTextData = [ 'a' => 123, 'b' => 'xyz', ]; $ch = curl_init("..."); $postData = [ 'data' => json_encode($someTextData), 'file1' => file_get_contents(__DIR__ . "/a.pdf"), ]; $postFiles = [ 'file1' => [ 'type' => 'application/pdf', 'content' => file_get_contents(__DIR__ . "/a.pdf")], ]; $delimiter = '-------------' . uniqid(); $data = ''; //region Populate file fields foreach ($postFiles as $name => $file) { $data .= "--" . $delimiter . "\r\n"; // "filename" attribute is not essential; server-side scripts may use it $data .= 'Content-Disposition: form-data; name="' . $name . '"' . '; filename="' . $name . '.pdf"' . "\r\n" // this is, again, informative only; good practice to include though . 'Content-Type: ' . $file['type'] . "\r\n" // if data is binary, I found it unnecessary but include it here . 'Content-Transfer-Encoding: binary' . "\r\n" . ''; // this endline must be here to indicate end of headers $data .= "\r\n"; // the file itself (note: there's no encoding of any kind) $data .= $file['content'] . "\r\n"; } //endregion foreach ($postData as $name => $content) { $data .= '--' . $delimiter . "\r\n"; $data .= 'Content-Disposition: form-data; name="' . $name . '"'; $data .= "\r\n\r\n"; $data .= $content . "\r\n"; } // last delimiter $data .= "--" . $delimiter . "--"; curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); // default is 2. In current version, 1 means the same as 2. Use 2 instead of 1. 0 means the connection succeeds regardless of the names in the certificate. 2 means that it has to have the same name in the certificate as is in the URL you operate against curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); // accept any server (peer) certificate. Default 1: enable CA checking. If this is 0, CURLOPT_SSL_VERIFYHOST will become 'does not matter'. curl_setopt( $ch, CURLOPT_HTTPHEADER, [ 'Content-Type: multipart/form-data; boundary=' . $delimiter, 'Content-Length: ' . strlen($data), ] ); curl_setopt($ch, CURLOPT_POSTFIELDS, $data); $rs = curl_exec($ch);
1.36.1.3. CURLOPT_ENCODING
- Set the request header
Accept-Encoding:and also enable decoding of the response- One single value
- "", "identity", "deflate", and "gzip"
1.36.1.4. POST with CA
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2); curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/CAcerts/BuiltinObjectToken-EquifaxSecureCA.crt");
1.36.1.5. POST with SSL Certificate and Key
Convert PFX to a private key
cl.pemand a certificatecl.cert. A passphrase is neededopenssl pkcs12 -in a.pfx -nocerts -out cl.pem -nodes openssl pkcs12 -in a.pfx -nokeys -out cl.crt -nodes # Test curl -v --key ./cl.pem --cert ./cl.crt https://xxx
PHP
$curl = curl_init(); $xml = ''; curl_setopt($curl, CURLOPT_URL, "https://xxx.ca/endpoint?a=xyz"); curl_setopt($curl, CURLOPT_HTTPHEADER, array( 'Content-Length: ' . strlen($xml), 'Content-Type: application/xml', )); $keyFile = 'cl.pem'; $certFile = 'cl.crt'; curl_setopt($curl, CURLOPT_POST, 1); curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); curl_setopt($curl, CURLOPT_POSTFIELDS, $xml); curl_setopt($curl, CURLOPT_SSLKEY, $keyFile); curl_setopt($curl, CURLOPT_SSLCERT, $certFile); curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, true); curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 2); curl_setopt($curl, CURLOPT_VERBOSE, 1); var_dump( curl_exec($curl)); var_dump( 'info', curl_getinfo($curl)); var_dump( 'error', curl_error($curl)); var_dump( 'errno', curl_errno($curl));
1.36.1.6. curl_error, http_code
$curl = curl_init(); $r['res'] = curl_exec($curl); $r['info'] = curl_getinfo($curl); $r['error'] = curl_errno($curl); $r['error'] = $r['error'] ? "curl_errno: {$r['error']} " . curl_error($curl) : ''; if ($r['info']['http_code'] !== 200) { // 0 :: could be timeout echo $r['error']; }
1.36.1.7. curl_getinfo()
// To see the request $ch = curl_init('https://abc.com'); curl_setopt($ch, CURLINFO_HEADER_OUT, true); $res = curl_exec($ch); echo "Request: " . PHP_EOL . curl_getinfo($ch, CURLINFO_HEADER_OUT) . PHP_EOL;
1.36.2. cURL-less
Use file_get_contents for PHP5. Some host might disable file_get_contents. Use cURL method Make sure php:ini:allow_url_fopen is enabled
$url = 'http://server.com/path'; $data = array('key1' => 'value1', 'key2' => 'value2'); // use key 'http' even if you send the request to https://... $options = array( 'http' => array( 'header' => "Content-type: application/x-www-form-urlencoded\r\n", 'method' => 'POST', 'content' => http_build_query($data) ) ); $context = stream_context_create($options); $result = file_get_contents($url, false, $context); if ($result === FALSE) { /* Handle error */ } // need to make sure string is UTF-8 php:string:encoding var_dump($result);
1.37. HTTP Header
-
header("Cache-Control: no-store, no-cache, must-revalidate, max-age=0"); header("Cache-Control: post-check=0, pre-check=0", false); header("Pragma: no-cache");
Set amount of time to cache
$seconds_to_cache = 3600; $ts = gmdate("D, d M Y H:i:s", time() + $seconds_to_cache) . " GMT"; header("Expires: $ts"); header("Pragma: cache"); header("Cache-Control: max-age=$seconds_to_cache");
Array
header('response: ' . json_encode($response));
var r = JSON.parse(rq.getResponseHeader('response')); if (r.status) { // ... }
headers_list()- Return all HTTP request headers
getallheaders()- alias of
apache_request_headers(). Only works for Apache web server- Earlier only works for PHP running in Apache as a module
- Later supported for CGI and later from PHP 7.3 supported for FGM
- (no term)
$_SERVER['HTTP_MY_REQUEST_HEADER']- Apache
- If PHP is not running as a Apache module, then it's available
- Any HTTP request header name will be converted to uppercase and dash to underscores
If PHP is running as an Apache module, do this
SetEnvIf MY_REQUEST_HEADER "(.*)" HTTP_MY_REQUEST_HEADER=$1
Don't echo before response header is set
// Don't do this echo "MESSI is injured!!"; header("Location:somepage.php"); // Instead, do this b_start(); echo "MESSI is injured!!"; header("Location:somepage.php"); ob_end_flush();
1.38. Throwable Interface
- https://www.php.net/manual/en/class.throwable.php
- public methods
- getMessage()
- getCode()
- getFile()
- getLine()
- getTrace()
- getTraceAsString()
- getPrevious()
- __toString
1.39. Exception
- https://www.php.net/manual/en/language.exceptions.php
- https://www.php.net/manual/en/class.exception.php
- Parent class
- see 1.38
- Constructor
public Exception::__construct(string $message = "", int $code = 0, ?Throwable $previous = null)
try { throw new Exception('Some message', 30); } catch (Exception $e) { // final public Exception::getMessage(): string $e->getMessage(); // final public Exception::getCode(): int $e->getCode(); }
1.40. Error Exception
- https://www.php.net/manual/en/language.errors.php7.php
- Parent class
- see 1.38
- (no term)
- Use
catch (Error $e) {...}, orcatch (Throwable $t) {...}orset_exception_handlerto handle fatal error- If an
Errorexception is nottry catchandset_exception_handleris not set, thenErrorexception becomes a fatal error, and later set_error_handler deals with the fatal error - PHP 7 now throws
Errorexceptions
- If an
- (no term)
- Child classes
- ArithmeticError
- AssertionError
- CompileError
- ParseError
- TypeError
- ArgumentCountError
- ValueError
- UnhandledMatchError
- FiberError
try { // throw new Exception('an exception'); // throwAFatalError(); // die does not throw fatal error nor exception die('throw a fatal error'); } catch (Exception $e) { echo "This is an exception: " . $e->getMessage(); } catch (Error $e) { echo "This is a fatal error: " . $e->getMessage(); }
1.41. Type declaration
- https://www.php.net/manual/en/language.types.declarations.php
- Can be added to
- function arguments
- return values
- class properties since PHP 7.4
- A child's method must match any return type declartion of the parent
- Parent may not define a return type, then the child method may do so
- Types
- void
- since PHP 7.1
- nullable type
- since PHP 7.1
?string- either a string or null
function f (C $c = null) {}- this is also possible but not recommended
self,parent- There's no
staticin PHP 7.x! An instance of the same class static- since PHP 8.0
mixed- since PHP 8.0
- Union types
- since PHP 8.0
string|null|int
- Intersection typen
- since PHP 8.1
- (no term)
- Some others
1.42. Pass by Reference
Don't easily pass by reference in
foreachloop$a = array ('zero','one','two', 'three'); foreach ($a as &$v) { } // $v is a reference of $a[3] foreach ($a as $v) { // $a[3] is changed to zero, one and two echo $v . PHP_EOL; } // zero one two two
1.43. Troubleshooting
1.43.1. Parse error: syntax error, unexpected end of file
Or Parse error: syntax error, unexpected 'endwhile' (t_endwhile)
Check if php short_open_tag is enabled. Default is not enabled. Then search <? ~ with a space and replace them with ~<?php
1.44. Naming Convention
- PSR Naming Conventions
- OOP
- ClassNamesLike
- MyTrait
- methodName
- propertyName
const CONSTANTS_LIKE_THIS
- global functions
- local functions
- global variables
- Global names must be prefixed or use namespace
$localVariableNamedefine('CONSTANTS_LIKE_THIS', '');- - Global names must be prefixed or use namespace
- File and directory
No standard but I prefer
- Directory
public_html
I would avoid using hypen and use nested directory structure
lawyer/lawyeronelawyer/laywertwo
- File
myclassname_is_long.class.php
1.45. Shell Exec Command
echo shell_exec("whoami");
1.46. Session
- PHP relies on client cookie
PHPSESSIDto identify session - js:xmlhttprequest made on the same domain always transfer all cookies
- php:ini:session
Need to start session. check if session is already started
if (session_id() == "") session_start();
1.47. Predefined Interfaces and Classes
1.47.1. IteratorAggregate and Iterator (interfaces)
interface IteratorAggregate extends Traversable { /* Methods */ public getIterator(): Traversable } interface Iterator extends Traversable { /* Methods */ public current(): mixed public key(): mixed public next(): void public rewind(): void public valid(): bool }
1.48. Composer
1.48.1. Install
1.48.1.1. Ubuntu
# php8.2-curl is required # apt install php8.2-curl # It's better to install zip/unzip (or 7z) rather than relying on php8.2-zip # apt install zip unzip # download installer, run it using PHP and put the resulted executable as in /usr/local/bin/composer curl -sS -L https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer composer --version
1.48.2. Basics
composer require abc/xyz- downloads the latest version to current folder
./vendor/composer/abc/xyz/*.*and add torequireincomposer.jsoncomposer require abc/xyz:0.3.*- specify a range of versions
- To use this package in PHP file
require 'vendor/autoload.php'; use Abc\Xyz\component1;
- (no term)
- Whether packages are installed locally or globally, it will use the user's directory
~/.composer. If current user is not root, don't run composer with sudo - (no term)
composer global require "abc/xyz"- global package is installed
- Installed path default is
~/.composer/vendor/bincomposer global config bin-dir --absoluteto find out the exact path- make sure the path is in PATH
echo 'export PATH="$PATH:/root/.config/composer/vendor/bin"' >> ~/.bashrcsource ~/.bashrc
- (no term)
composer global remove abc/xyz- (no term)
- composer.json
composer init- initialize proj and create composer.json
composer install- install packages based on
composer.json - (no term)
require"abc/packagename": "0.3.*"- ">~1.2 eq. to >=1.2 <2.0.0, ~1.2.3 is eq. to >=1.2.3 <1.3.0
- ^1.2.3 is eq. to >=1.2.3 < 2.0.0
- (no term)
require-dev- added by
composer require --dev
- added by
- (no term)
composer outdated- (no term)
composer updateorcomposer update abc/pkgnamecomposer show- show installed packages
- (no term)
composer remove [vendor/package]- (no term)
composer self-updatecomposer -V
.gitignore/vendor/
1.48.3. composer.json
- autoload
- after edit, run
composer dump-autoload -opsr-4
"autoload": { "psr-4": { "TDD\\": "src/" } },
- Map a namespace TDD to directory
src "psr-4": { "Monolog\\": ["src/", "lib/"] }"psr-4": { "": "src/" }
- Map a namespace TDD to directory
build map by scanning for classes in all
.phpand.incfiles in specified directories/files{ "autoload": { "classmap": ["src/", "lib/", "Something.php"] } }
1.48.4. Packages
- PHP Package Repo
- https://packagist.org/
1.48.4.1. PHP Code Sniffer
- AKA phpcs, PHP CodeSniffer, PHP Code Sniffer
- Static code analysis for PHP, JavaScript and CSS files
- https://packagist.org/explore/?type=phpcodesniffer-standard
- wp-coding-standards/wpcs
- phpcompatibility/php-compatibility
1.48.4.2. squizlabs/php_codesniffer
- https://github.com/squizlabs/PHP_CodeSniffer
- https://github.com/squizlabs/PHP_CodeSniffer/wiki/Usage
composer global require "squizlabs/php_codesniffer=*" # Make sure composer bin dir in PATH # Check the composer bin dir location # default should be ~/.composer/vendor/bin/ # - 2 files are installed under this directory: phpcs and phpcbf # - ~/.composer/vendor/squizlabs/php_codesniffer/src/Standards/PSR12 is the installation path for standard PSR12 composer global config bin-dir --absolute # export PATH="$PATH:$HOME/.composer/vendor/bin" # My docker container is /root/.config/composer/vendor/bin echo 'export PATH="$PATH:/root/.config/composer/vendor/bin"' >> ~/.bashrc source ~/.bashrc phpcs --version # see all installed coding standards # PSR12 is installed by default phpcs -i # List all sniffers included in a standard phpcs --standard=PSR12 -e cd /path/to/my-project phpcs --standard=PSR12 sample.php
1.48.4.3. phpcodesniffer-composer-installer
# https://github.com/PHPCSStandards/composer-installer # It can install phpcs and also standards composer require --dev dealerdirect/phpcodesniffer-composer-installer
Sample
{ "require-dev": { "dealerdirect/phpcodesniffer-composer-installer": "^0.4", "wp-coding-standards/wpcs": "^2", "phpcompatibility/php-compatibility": "^9" }, }- And run
composer install - PHP_CodeSniffer is installed at
./vendor/bin/phpcsand it links to multiple standards installed above- Set it as PHP_CodeSniffer phpstorm:php:quality tools
1.49. PHPUnit
- https://phpunit.de/getting-started/phpunit-9.html
- v9
- PHP 7.3, 7.4
- v8
- PHP 7.2, 7.3
composer require --dev phpunit/phpunit ^9./vendor/bin/phpunit --version
composer.json should look like
{ "autoload": { "classmap": [ "src/" ] }, "require-dev": { "phpunit/phpunit": "^9" } }- Code to test
src/Email.php, Test file istests/EmailTest.php. A test is a public method namedtest*or add@testannotation in a method's docblock ./vendor/bin/phpunit tests- Run one test file
./vendor/bin/phpunit tests/ReceiptTest.php- Filter test cases in files and run tests
./vendor/bin/phpunit tests --filter=testTax- Run function that matches the regex
testTax - e.g. matched methodName
- (no term)
./vendor/bin/phpunit tests --filter=ReceiptTest::testTax
- Old version of PHPUnit
phpunit --filter testTax path/to/ReceiptTest.php
- Run function that matches the regex
- Filter by testsuite
./vendor/bin/phpunit --testsuite=app --filter=testTax
- https://phpunit.readthedocs.io/en/8.5/assertions.html
assertEquals(mixed $expected, mixed $actual[, string $message = ''])
- phpstorm:phpunit
1.49.2. CLI
- Syntax
phpunit --helpphpunit [options] UnitTest.phpphpunit [options] <directory>
- Characters to indicate progress
- .
- success
- F
- fail
- E
- error
- R
- marked as risky
- S
- skipped
- I
- incomplete or not yet implemented
- Options of config
-d key[=value]- set a php.ini value e.g.
-d memory_limit=256M --configuration,-c- e.g.
-c sub/phpunit.xmlread config. If omitted,phpunit.xmlorphpunit.xml.distat current dir. will be read
- Options of test selection
--testsuite <name,pattern,...>- which testsuites to run
- Options of Test Execution
--debug
1.50. Standard PHP Library (SPL)
1.50.1. Interfaces
1.50.1.1. SplObserver
- See Observer Design Pattern
- Functions
- update
1.50.1.2. SplSubject
- See Observer Design Pattern
- Functions
- attach
- detach
- notify
1.51. Libraries
1.51.1. mPDF
Convert HTML to PDF https://github.com/mpdf/mpdf https://mpdf.github.io/css-stylesheets/supported-css.html
1.51.1.1. @page
https://mpdf.github.io/paging/using-page.html
@page { margin-top: 1cm; margin-bottom: 3cm; margin-left: 2cm; margin-right: 2cm; } body { background-image: url(paper-size-ratio.jpg); background-repeat:no-repeat; background-position:top center; background-image-resize:6; }
Paper size
- A4
- 210 × 297 millimeters or 8.27 × 11.69 inches
1.51.2. Mobile-Detect
Device tect, browser detect :: GitHub Examples
require_once 'Mobile_Detect.php'; $detect = new Mobile_Detect; $detect->isIphone(); $detect->isSamsung(); $detect->is('iphone'); $detect->version('Android'); if ($detect->isMobile()) {} if ($detect->isTablet()) {} // Check for any mobile device, excluding tablets. if ($detect->isMobile() && !$detect->isTablet()) {}
1.51.4. Laravel
1.51.4.1. Installation
- A skeleton Laravel project runs on top of the Laravel Framework
- https://github.com/laravel/laravel
composer create-project laravel/laravel example-app
- Laravel Framework
- https://github.com/laravel/framework
- (no term)
chmod -R 777 storage
- Nginx
server { server_name my.website.com; root /var/www/mywebsite/public; add_header X-Frame-Options "SAMEORIGIN"; add_header X-Content-Type-Options "nosniff"; index index.php; charset utf-8; location / { try_files $uri $uri/ /index.php?$query_string; } location = /favicon.ico { access_log off; log_not_found off; } location = /robots.txt { access_log off; log_not_found off; } error_page 404 /index.php; location ~ \.php$ { fastcgi_pass unix:/var/run/php/php8.2-fpm.sock; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; include fastcgi_params; } location ~ /\.(?!well-known).* { deny all; } listen 80; }
1.51.4.2. Sail
- Installation (Laravel Framework 9.9.0)
curl -s https://laravel.build/example-app | bash cd example-app # Interactive mode for debugging ./vendor/bin/sail up # Detached mode ./vendor/bin/sail up -d # stop all running containers ./vendor/bin/sail stop # Get status of containers (remove exited containers) ./vendor/bin/sail status # nano ./.env # Add APP_PORT=81 # To solve Bind for 0.0.0.0:80 failed: port is already allocated # Then go to http://localhost:81 # Laravel Source Code cd ./vendor/laravel/framework/src/ # e.g. ORM cd Illuminate/Database/Eloquent/
- CLI
# Display Help ./vendor/bin/sail # Container CLI sail shell # eq. sail bash sail root-shell sail root-bash # Artisan commands sail artisan --version
- .env
1.51.4.3. Artisan CLI
php artisan --version # List of all commands php artisan list cd /path/to/my/website # manually modify .env file # whenever .env file is created or updated php artisan config:cache # generate key, .env > APP_KEY is modified php artisan key:generate php artisan config:cache cd database/seeders php artisan migrate # .env > JWT_SECRET is inserted php artisan jwt:secret php artisan config:cache
1.51.4.5. Architecture
- Short Version
- create an app
- attach singleton of kernals: like api, http kernels
- Instantiate the kernal
- Ask kernal to handle a request
- Kernal runs
registerfor each service provider defined in config/app.php- Attach/bind more things to the $app (dependency injection)
- Instances of Service Container
- Facade, which can be:
- singleton of a service container
- singleton of any class!
To access
$requestin Service Provider'sregisterhas no access$request// Contextual Binding public function register() { $this->app->when(Service::class) ->needs(Thingable::class) ->give(function () { return request()->user_name === 'userA' ? $this->app->get(DependencyA::class) : $this->app->get(DependencyB::class); }); } // or public function register() { $this->app->bind('App\Thingable', function ($app) { return $app->make(request()->user_name === 'userA' ? 'App\DependencyA' : 'App\DependencyB'); }); }
- Attach/bind more things to the $app (dependency injection)
- Kernal runs
bootfor each service provider - Kernal runs all middlewares which are defined when service providers are registered
- Kernal deals with the request
- Pick a route that is defined at
routes/web.php- Instantiates the corresponding controller defined in the route
- And do the method of the controller e.g. show loan, delete loan
- Either use the previously injected dependencies (from the $app)
- or use the Facade
- Pick a route that is defined at
- Kernal runs all middlewares which are defined when service providers are registered, run them before delivering the response
- Kernal delivers the response
- Kernal runs
- Return the response after dealing a request
- Terminate the kernel
- create an app
- Long Version
- public/index.php
- bootstrap/app.php
- create an instance of the application (service container which extends
Illuminate\Container\Container)- Request is sent to either the HTTP kernel (
app/Http/Kernel.php) or the console kernal.- HTTP kernel's
handlemethod is simple: receiveRequestand returnResponse- Think of the kernel as being a big black box that represents the entire app.
- HTTP kernel extends
Illuminate\Foundation\Http\Kernel bootstrappersare defined and run before the request is executed- config error handling
- config logging
- detect app environment
- perform other tasks that need to be done before the request is handled
- One of the most important kernel bootstrapping actions is loading the Service Providers
- Service providers are configured in
configure/app.php, theprovidersarray- Each provider is instantiated and the corresponding methods are called:
registermethod is calledbootmethod
- Each provider is instantiated and the corresponding methods are called:
- Routing is a service provider
App\Providers\RouteServiceProviderloadsroutesdirectory- Once the app is bootstrapped and all service providers are registered,
Requestwill be handled off to the router for dispatching. The router will dispatch the request to a route or controller, as well as run any route specific middleware. - After all the route's assigned middleware has passed through, the route or controller method will be executed and the response returned by the route or controller method will be sent back through the route's chain of middlewares
- Once the route or controller method returns a response, the response will travel back outward through the route's middleware, giving the app a chance to modify or examine the outgoing response.
- Finally, once the response travels back through the middleware, the HTTP kernel's
handlemethod returns the response object and theindex.phpcalls thesendmethod on the returned resposne. Thesendmethod sends the response content to the user's web browser.
- Service providers are configured in
- Middlewares are defined
- all requests must pass through before being handled by the application
- read and write HTTP session
- determine if the app is in maintenance mode
- verify CSRF token and more
- determine if the user is logged in
- all requests must pass through before being handled by the application
- HTTP kernel's
- Request is sent to either the HTTP kernel (
- create an instance of the application (service container which extends
- bootstrap/app.php
- public/index.php
1.52. Run PHP Online in multiple versions
1.53. Coding Standard
1.53.1. Variable, action/filter and function name
- all lowercase, never
camelCase - Separate words by underscore
1.53.2. Class name and file name
- Class name
- WordPress
- Capitalized words separated by underscores
- Acronym should be all upper case
- General PHP
UpperCamel- can have underscores and numbers
- (no term)
SampleXmlClass, notSampleXMLClass- (no term)
- Names should not include Drupal, Class
- (no term)
- Interface should always prefix
Interface
- WordPress
- Class file name
- lowercase with
class-preppended WP_Errorisclass-wp-error.php
- lowercase with
1.53.2.1. State (Behavioral DP)
1.53.2.2. Aggregation vs Composition
- Aggregation
- object A contains objects B; B can live without A.
- Composition
- object A consists of objects B; A manages life cycle of B; B can’t live without A.
2. Drupal
2.1. Install Drupal on Windows
2.2. Update Core
- pantheon:drupal:core
For Windows, you may need to stop and do the file changes and then start again. Restart IIS
iisreset -stop iisreset -start # this stops and starts iisreset
- https://www.drupal.org/docs/7/updating-your-drupal-site/how-to-update-drupal-core
- Summary
- Make backup
- Download and Extract
- Set website on maintenance mode
- Delete all files and folders
- don't delete folder
/sites - delete folders
/profiles/minimal/profiles/standard/profiles/testing - delete folders
/includes/misc/modules/scripts/themes - reapply
.htaccessrobots.txtweb.configsettings.php - reapply any modified files and folders
- don't delete folder
- Some updates don't include settings.php. If it includes, replace the old one in sites/default with the new one and apply your previous changes
- replace favicon.ico
- Login as user 1 and run update.php
- If you are unable to access update.php do the following: (allow update.php to be run when not logged in as admin)
- Open settings.php with a text editor.
- Find the line that says:
$update_free_access = FALSE;Change it into:$update_free_access = TRUE; - Try again to run update.php.
- Once the update is done, $update_free_access must be reverted to FALSE.
- In a multi-site installation, run update.php again for each site.
- Navigate to Administration > Configuration > Development > Performance and clear all cache
- Admin > Reports > Status to Verify everything is working as expected.
- Ensure that $update_free_access is FALSE in settings.php.
- Disable maintenance mode
https://www.drupal.org/project/module_missing_message_fixer to overcome these errors
The following module is missing from the file system: MODULE NAME. In order to fix this, put the module back in its original location. For more information, see the documentation page.
Or
User warning: The following module is missing from the file system: MODULE NAME. In order to fix this, put the module back in its original location. For more information, see the the documentation page. in _drupal_trigger_error_with_delayed_logging()
- In order to check available updates for core and modules, module Update Manager needs to be enabled
2.2.1. 7.x
Legend
- If no changes for .htaccess, web.config, robots.txt nor settings.php, then there's no description. Otherwise, it will indicate which ones to update.
7.50 :: d7:ts:module is missing robots.txt, default.settings.php, add .editorconfig file 7.51, 7.52 7.53 :: update jQuery to 1.7-1.11.0 7.54 7.55 7.56 :: security 7.57 :: security 7.58 :: security
2.2.2. Troubleshoot
2.2.2.1. The following module is missing from the file system d7:ts:module is missing
- https://www.drupal.org/node/2487215
- My situation is that a module is installed but removed from file system without disabling and uninstalling it
Because the module code no longer exists, so I need to manually remove traces in db. D7
drush sql-query "DELETE from system where name = 'old_module1' AND type = 'module';"
If module name is not found in db, most likely the code incorrectly refers to a non-existing module such as drupal_get_path('module', 'deletedmodname'). Do backtrace to identify.
2.3. Update Module
- Read module README first
- Usually just remove the module directory and put the new one in
- Run update.php
2.4. Module
2.4.1. Core Modules
2.4.1.1. Path d7:proj:path
- Set the path for an individual node with the Path module (on the node/add or node edit form)
- Add an URL alias at: Administer > Configuration > Search and metadata > URL aliases
- Administer the list of URL aliases at: Administer > Configuration > Search and metadata
- Administer > Configuration > Clean URLs
- https://www.drupal.org/docs/7/core/modules/path/overview
- May use d7:proj:pathauto
2.4.1.2. Entity API d7:entity
- Sub module
- entity_token
- (no term)
- Requires nothing
- (no term)
- Update
- 1.5
- 1.9 no db
2.4.2. Custom Module
2.4.2.1. .info file
- Instructions
- Module name must contain only lower-case and undersocre
sites/all/modules/custom/my_modulesites/all/themes/custom/my_thememy_module.infomy_module.module
- Use
;at the beginning of a line to mark as comment name = A Human Readable Namedescription = More is <a href="">here</a>.Only one line with 255 characters and it's HTML (e.g. accented characters)!core = 7.xCan't specify minor versionpackage = Viewspackage = Example modules- If your module comes with other modules or it's meant to be used exclusively with other modules, provide the package name here.
- More than 4 modules that depend on each other can make a package.
files[] = includes/context.incThis .inc file should be class or interface. To include .inc with plain functions- Clear cache when .info is changed
- A submodule under a package may not add parent module to dependencies and submodule names can be anything given the parent module is
examples:- action_example
- action_examples
- example_action
dependencies[] = entity dependencies[] = views (>=3.12) dependencies[] = exampleapi (1.x) dependencies[] = exampleapi (>7.x-1.5) dependencies[] = exampleapi (>1.0, <=3.2, !=3.0) dependencies[] = system (>=7.53)
Optional
configure = admin/config/content/my_mymodule ; hide on modules page. e.g. SimpleTest where end-users should never enable the testing modules hidden = TRUE
2.4.2.2. .install File
- Define new tables, load data and implement conversions during updates
- It's run the first tiem a module is enabled and is used to run setup procedures as required by the module
- No syntax. It's a PHP file
- https://www.drupal.org/docs/7/creating-custom-modules/writing-install-files-drupal-7x
called when the module is first enabled. Typical use is to create the necessary tables d7:hook_install
function your_module_name_install() { // change module weight db_update('system') ->fields(array('weight' => 999) ->condition('name', 'your_module_name', '=') ->execute(); // 1 heavier than another module's weight // Get the weight of the module we want to compare against $result = db_select('system', 's') ->fields('s', array('weight')) ->condition('name', 'the_other_module_name', '=') ->execute(); $weight = !empty($result) ? $result->fetchField() : 0; // Set our module to a weight 1 heavier, so ours moves lower in execution order db_update('system') ->fields(array('weight' => $weight + 1)) ->condition('name', 'your_module_name', '=') ->execute(); }
hook_schema()- will be called by
update.php- 1 digit for Drupal core compatibility
- 1 digit for module's major release version. Use 0 for initial porting of the module to a new Drupal core API
- 2 digits for sequential counting, starting 00
- The required update for mymodule to run with Drupal core API 7.x when upgrading from 6.x
- The first update to get the db ready to run mymodule 7.x-1.*
- The first update to get the db ready to run mymodule 7.x-2.*. Users can directly update from 6.x-2.* to 7.x-2.* and they get all 70xx and 72xx updates, but not 71xx updates, because those reside in the 7.x-1.x branch only
2.4.2.3. Refactor, include .inc files
function ctools_include($file, $module = 'ctools', $dir = 'includes') { static $used = array(); $dir = '/' . ($dir ? $dir . '/' : ''); if (!isset($used[$module][$dir][$file])) { require_once DRUPAL_ROOT . '/' . drupal_get_path('module', $module) . "$dir$file.inc"; $used[$module][$dir][$file] = TRUE; } } // ctools_include('utility', 'ctools', 'includes'); // ctools_include('macro', 'mymodule', 'includes');
2.4.2.4. lili.api.php
Just include this in module root directory and all hooks will be defined
2.4.3. sqlsrv
Install PDO Microsoft Drivers for PHP for SQL Server https://docs.microsoft.com/en-us/sql/connect/php/system-requirements-for-the-php-sql-driver?view=sql-server-2017
For PHP 5.x, use PDO 3.2 For PHP 7.x, use PDO 5.2
PDO 4.x and 5.x supports PHP 7 only. https://docs.microsoft.com/en-us/sql/connect/php/system-requirements-for-the-php-sql-driver?view=sql-server-2017#driver-versions
Download 4.0, 3.2, 3.1, 3.0 https://www.microsoft.com/en-us/download/details.aspx?id=20098 run SQLSRV32.EXE on the real server, select the version and any place to extract the drivers.
To download other versions for PHP 7 :: https://docs.microsoft.com/en-us/sql/connect/php/download-drivers-php-sql-server?view=sql-server-2017
To find out which PHP version is used in IIS :: Your website > Handler Mappings > FastCgiModule Setup PHP on IIS https://docs.microsoft.com/en-us/iis/web-hosting/web-server-for-shared-hosting/fastcgi-with-php
If php5.dll is used, then it's non-threaded, if php5ts.dll (thread safe), then it is threaded. Use the correct version of the SQLSERV and PDO_SQLSERV drivers.
Non-thread safe :: php_sqlsrv_54_nts.dll php_pdo_sqlsrv_54_nts.dll
Put those into ext folder. Check extension_dir in php.ini.
Change these lines in php.ini
;extension=php_pdo_sqlsrv.dll ;extension=php_sqlsrv.dll extension=php_pdo_sqlsrv_54_nts.dll extension=php_sqlsrv_54_nts.dll
Run iisreset to restart IIS.
2.4.4. Pathauto d7:proj:pathauto
- The Pathauto module automatically generates URL/path aliases for various kinds of content (nodes, taxonomy terms, users) without requiring the user to manually specify the path alias. This allows you to have URL aliases like /category/my-node-title instead of /node/123. The aliases are based upon a "pattern" system that uses tokens which the administrator can change
- Requires
- d7:proj:path
- d7:proj:token
2.4.5. Feeds - Package: Feeds
Requires d7:ctools d7:job_scheduler Test requires: date:date entity_translation:entity_translation feeds_xpathparser:feeds_xpathparser i18n:i18n_taxonomy link:link rules:rules variable:variable Submodules:
- feeds_import
- feeds_news
- feeds_ui
$feed_obj = feeds_source($feed_id,$feed->feed_nid); $feed_obj->existing()->import(); // Use this so no jobs are inserted in batch // Sequence $this = feeds_source; hook_feeds_before_import($this) hook_feeds_after_parse($this, $parser_result) FeedsProcessor.inc->process parser_result releaseLock() hook_feeds_after_import($this)
Import drush command runtime can be long. Output something every 5 minutes to prevent drush command timeout Do this in one of the hooks:
- hook_feeds_before_update
- hook_feeds_presave
- hook_feeds_after_save
function lili_feeds_presave(FeedsSource $source, $entity, $item) { $request_time = lili_custom_cache('timecounter'); $interval = 60 * 5; if (!is_null($request_time)) { $row_counter = lili_custom_cache('rowcounter'); lili_custom_cache('rowcounter',++$row_counter); $current_time = time(); $seconds = $current_time - $request_time; if (floor($seconds/$interval) > 0) { lili_custom_cache('timecounter', $current_time); print "Presave row #$row_counter. $seconds seconds have passed.\n"; } } else { lili_custom_cache('timecounter', REQUEST_TIME); lili_custom_cache('rowcounter', 1); } }
2.4.5.1. New Feed Tamper Plugin
The plugin does this: Drupal download file with real file extension
// Implements hook_ctools_plugin_directory() function lili_ctools_plugin_directory($module, $plugin) { if ($module == 'feeds_tamper') { return 'plugins'; // path_to_lili_module/plugins/lili_*.inc } } // lili_change_image_extension.inc // You can refer to path_to_feeds_tamper_module/plugins/explode.inc $plugin = array( 'form' => 'lili_image_extension_form', 'callback' => 'lili_image_extension_callback', 'validate' => 'lili_image_extension_validate', // optional 'name' => 'Convert Image URL Extension', 'category' => 'T and T', // group name 'multi' => 'direct', // If multiple values are injected to the plugin, // 'direct' will pass the whole array as $field to callback // while 'loop' will loop over and pass each value to callback ); function lili_image_extension_form($importer, $element_key, $settings) { $form = array(); $form['info'] = array( '#markup' => t('Converts source image URLs that don\'t have regular image extension: jpg, png, etc.'), ); // Extra variables to pass to $settings in callback $form['importer_name'] = array( '#type' => 'hidden', '#title' => t('Importer machine name'), '#default_value' => isset($settings['importer_name']) ? $settings['importer_name'] : $importer->id, ); return $form; } function lili_image_extension_validate(&$settings) { // Validate $settings. } function lili_image_extension_callback($result, $item_key, $element_key, &$field, $settings, $source) { //settings has the importer name //$settings['importer_name'] //$field has the importing field text value if (!is_array($field)) { $field = array($field); } $val = array(); foreach ($field as $f) { $val[] = _lili_save_file_with_extension($f); } $field = $val; }
2.4.5.2. XPath
Children nodes Photo under Images node.
Simply Images/Photo, the result is already array
Concate multiple nodes
concat(photo1,';',photo2) then explode
2.4.6. Feeds Tamper - feeds_tamper - Package: Feeds
Requires feeds Submodule: feeds_tamper_ui
2.4.7. Feeds Tamper Importer - feeds_tamper_importer - Package: Feeds
Requires feeds_tamper
2.4.8. Feeds XPath Parser - feeds_xpathparser - Package: Feeds
Requires feeds
2.4.9. Feeds entity processor - feeds_entity_processor - Package: Feeds
Requires feeds d7:entity
2.4.10. XML Sitemap
admin/config/search/xmlsitemap/settings Uncheck /Prefetch URL aliases during sitemap generation
2.4.11. ThemeKey d7:module:themekey
You can switch to a different based on some rules! If you are not clear about some rules, you can just search in code. e.g. search 'system:query_string'
2.4.12. Devel
- requires nothing
- devel_generate, devel_node_access (off)
http://ratatosk.net/drupal/tutorials/debugging-drupal.html
dpm($input, $name = null) use drupal_set_message and Krumo to dispaly in 'message' area kpr($input, $return = FALSE, $name=NULL) uses Krumo print in page header not 'message' area. dvm($input, $name = NULL) use Krumo to var_dump in 'message' area. Good for copy and paste dpr($input, $return = FALSE, $name = NULL) print in page header not in 'message' area.
2.4.13. Administrator menu - admin_menu - Package: Administration
Requires nothing Submodules: admin_devel, admin_menu_toolbar
2.4.14. Administration Views - admin_views - Package: Administration
Requies d7:views d7:views_bulk_operations
2.4.15. Administer Users by Role - administerusersbyrole
Requires chain_menu_access
Allow users to edit/delete other users
2.4.16. Variable
variable requires nothing d7:m:variable
2.4.17. Libraries
requires nothing
2.4.18. Module Filter
module_filter :: requires nothing
2.4.19. Geocoder
Requires d7:geophp d7:ctools d7:entity d7:geocoder An API and widget to geocode text field into GIS data types.
2.4.20. Entity View Modes - entity_view_mode
Requires nothing d7:entity_view_mode Recommends d7:field_ui Create custom View Mode: Default, Teaser under Manage Display d7:ds can also create view mode
2.4.21. wysiwyg
requires nothing Allows the use of client-side editors to edit content 7.x-2.2 to 7.x-2.4 (support TinyMCE from 3.3.9.2 to 4.5.7)
2.4.21.1. Installation & Config
Setup assign text format to trusted roles: admin/config/content/formats Follow README.md to config Text Format (e.g. Full HTML). Basically remove any restrictions. Installation Instructions on config page: /admin/config/content/wysiwyg TinyMCE installation instructions are missing.. Refer to this https://www.drupal.org/docs/7/modules/wysiwyg/installation Basically, download from TinyMCE and put it in sites/all/libraries/tinymce
If TinyMCE editor is upgraded, make sure the options for the Wysiwyg profile using TinyMCE is the same as before admin/config/content/wysiwyg/profile e.g. https://www.todaystrucking.com/admin/config/content/wysiwyg/profile/full_html/edit You need to save the profile after the TinyMCE editor is updated.
Supported editor versions: https://www.drupal.org/project/wysiwyg
- All TinyMCE settings
To config other TinyMCE settings and the Wysiwyg module UI doesn't provide, use hook_wysiwyg_editor_settings_alter()
- HTML elements are stripped off
In /admin/config/content/wysiwyg/profile/full_html/edit, keep Verify HTML on.
function lili_wysiwyg_editor_settings_alter( &$settings, $context ) { dpm($settings, 'settings'); dpm($context, 'context'); $settings['extended_valid_elements'] .= ',script[language|type|src]'; }
- verify_html (bool)
- clean up HTML elements that are allowed for
valid_elements,extended_valid_elementsandinvalid_elements - valid_elements (string)
- there're some but <script> is not allowed
- HTML elements are stripped off
2.4.21.2. Troubleshoot
Fields in node or block configuration might be empty or some elements are missing after save. To see the HTML, disable javascript in Chrome.
2.4.22. IMCE Wysiwyg bridge: imce_wysiwyg
https://www.drupal.org/project/imce_wysiwyg 7.x-1.0 requires imce and wysiwyg
2.4.23. IMCE
requires nothing 7.x-1.10 to 7.x-1.11 IMCE is an image/file uploader and browser that supports personal directories and quota.
2.4.24. ckeditor
requires nothing
2.4.25. nodeblock
create a block for a node
2.4.26. Block Group
blockgroup :: requires core Block d7:blockgroup This module extends the standard drupal block system with block groups. Each block group provides a new block as well as a corresponding region. Child blocks can be moved into any group region. The position and the settings of the parent block are propagated to its children. Also block groups are nestable.
2.4.27. MultiBlock d7:multiblock
- multiblock
- Built-in in D8. Requires core Block
2.4.28. Conditional Fields
conditional_fields :: d7:module:conditional fields
Conditional Fields for Drupal 7 is an user interface to the new States API, plus the ability to modify fields appearance and behavior on certain conditions when viewing content.
Conditional Fields allows you to manage sets of dependencies between fields. When a field is “dependent”, it will only be available for editing and displayed if the state of the “dependee” field matches the right condition. When editing a node (or any other entity type that supports fields, like users and categories), the dependent fields are dynamically modified with the States API. A simple use case would be defining a custom “Article teaser" field that is shown only if a "Has teaser" checkbox is checked, but much more complex options are available.
It works on AJAX node edit/new form
2.4.29. Search404
search404 :: requires core search
If a user goes to http://example.com/does/not/exist, this module will do a search for "does not exist" and shows the result of the search instead of the 404 page.
2.4.30. Google Analytics
google_analytics :: requires nothing
2.4.31. Elysia Cron
elysia_cron :: Requires nothing. Don't put a cron key in Elysia otherwise the cron run in drush will not work.
2.4.32. SMTP Authentication Support
smtp :: requires nothing
2.4.33. Chaos Tools - ctools
2.4.34. Address Field - addressfield - Package: Fields
Requires d7:ctools
Field List: admin/reports/fields
/moduels/field/field.api.php https://api.drupal.org/api/drupal/modules%21field%21field.api.php/7.x
2.4.35. Email - email - Package: Fields
Requires nothing
2.4.36. Geofield - Package: Fields
Requires d7:geophp d7:ctools Used with d7:geocoder
2.4.37. Viewfield - Package: Fields
Requires d7:views e.g. An order has multiple items, store those items into a field
2.4.38. Field collection - Package: Fields d7:proj:field_collection
2.4.39. Social Field - socialfield - Package: Fields
Requires nothing
2.4.40. Video Embed Field - video_embed_field - Package: Media
Requires d7:ctools core image New field type Used with video_embed_* modules
Have to install other modules to support different providers e.g. video_embed_facebook, video_embed_brightcove
2.4.41. Display Suite - ds - Package: Display Suite
Requires d7:ctools d7:ds Submoduels
- ds_devel
- requires devel
- ds_extras
- admin/structure/ds/list/extras
- Field Templates
- theme a field. To theme a display field, you don't need to enable it
- (no term)
- Extra Fields
- (no term)
- Other
- Region to block
- Create a region and move fields into that region Content Type > Manage Display > Block regions This will create a block which can be used in other places/regions in Blocks setting
- View mode per node
- specify a view mode for each node
- (no term)
- ds_format
- (no term)
- ds_forms
- (no term)
- ds_search Integrate search engine such as module apachesolr and d7:search_api_solr to display the search results for a content type
- (no term)
- ds_ui
https://www.youtube.com/playlist?list=PL7E361A55994F1648
Can display fields of a content type into a layout which has several regions under Manage Display
A display field can be created for a content type display. This field is for theming only not physically create a field in db. Field value can be PHP code and Token can be used.
- Field
- custom field with PHP and Token
- Block field
- display a block
- Dynamic field
- choose a variable to display e.g. node_body, menu
- Preprocess field
- display a variable e.g. Node Url (machine name node_url)
Later theme that display field using ds_extras
View Mode :: Default, Teaser Create a view mode. d7:entity_view_mode can also create view modes. To display a view that returns multiple nodes of a content type, go to Format > Show > Display suite > Settings Choose the custom view mode and also specify the first item to use View Mode #1 and the second item to use View Mode #2, etc. Custom hook can be selected in this setting to make customization
2.4.42. Meta Tag - metatag
- requires ctools and token
- Add metadata tags such as meta description, meta keywords, social media Open Graph
2.4.43. Entity Reference - entityreference
entityreference :: requires d7:ctools d7:entity d7:entityreference In D8 core Add a entity reference field which you can use an entity reference view as a list of values to select from as values in that field. Refer to d7:entity reference view
2.4.44. File Entity
file_entity d7:file entity Requires: d7:ctools, core: Field, Field SQL storage, File
From 7.x-2.0-beta2 to 7.x-2.3
2.4.45. Webform d7:webform
2.4.46. Webform Validation
webform_validation :: requires d7:webform
2.4.47. Panels
2.4.48. Facet API d7:facetapi
- Requires d7:ctools
- current_search, facetapi_bonus
2.4.49. Search API - search_api
Requires entity d7:search_api sub modules :: search_api_views (views), search_api_facetapi (d7:facetapi) https://www.drupal.org/docs/7/modules/search-api https://www.youtube.com/watch?v=hcAM0HrEk4c
2.4.49.1. Glossary
For example, a "Node index" for indexing nodes. It would contain the fields that should be indexed and their types, the data alterations and processors to use and some other settings. The details of how to index data are independent of these settings, and server-specific. Index settings are independent of the inner mechanics of the search server.
A server is a concrete way to index and search data. It could, e.g., represent a certain database, a connection to an external search server, etc. How exactly the data is stored is determined by the server's service class, but is not important for the overall functionality of the Search API. A server can have an arbitrary number of indexes attached to it, whose search data is then indexed on that server.
When creating a server, a service class has to be chosen
2.4.49.2. Index
If an index is updated, reindex all content.
- Select index fields
Indexed for those fields for which you want to store data on the search server. These fields can then be searched, used for filtering and sorting, and maybe also used for other purposes.
Note that only fields of type Fulltext can be used in fulltext searches. So when you want to find individual words contained in this field, not just the whole field value, use this type. Other types can be used, e.g., for filtering and sorting.
Fields indexed with type "Fulltext" and multi-valued fields (marked with 1) cannot be used for sorting. The boost is used to give additional weight to certain fields, e.g. titles or tags. It only takes effect for fulltext fields.
- Add Related Fields
Items of a certain type might be connected to, or might reference, other kinds of data. For instance, content will always have an author, could also contain references to taxonomy terms, etc. With the Add related fields form at the bottom of the page you can add the fields of those related items to the list, so they can be indexed, too. Use this if you want, e.g., to index the user roles of a node's author. You can also add nested related fields, e.g., the node's author's profile's image's file type.
- Customize Workflow (Filters Tab)
- Data alterations
Bundle filter Lets you to prevent entities from being indexed based on their bundle (content type for nodes, vocabulary for taxonomy terms, etc.). This way you can, for instance, create an index solely for news. Language control Allows you to control the language of items stored in the index. This is done by providing two different functionalities: Normally, the content of the Item language property (which is automatically added by the Search API for all indexed items) is determined by the item's language property, if available, and otherwise set to undefined. With this data alteration, you can select any other property as an alternative source for the item language, which will then be used instead. Note that the selected field has to contain a single valid ISO language code for each item for this to work, though. You can then also select the languages items in this index may have. Items with any other language (defined by the Item language property) will be rejected during indexing.
Node access Adds node access checks to searches on this index. This is done by adding a new field, Node access information that stores the relevant access data. When the Node access information, author, and Status fields are present and indexed, appropriate filters will be automatically added to all searches so that they only return results that the current user is allowed to view. Some searches (e.g., search views) provide the option to override this behaviour on a per-search basis, though. Check the corresponding module's documentation for details. In any case, you have to keep in mind that these access checks are solely based on the indexed data. If a node is edited in a way that changes its accessibility (e.g., by being unpublished), this change will only take effect once the node is indexed in its latest state. This means that there is potentially a gap between changing the node and the update of the access checks on search results, meaning that—depending on the data displayed for search results—users could in that time see data that should not be accessible to them. If you need to avoid that, use the index's Index items immediately option.
Also note that access on the individual fields is never checked — don't include them in the display, if they contain sensitive data. Refer to hook_node_access_records() and hook_node_grants() on implementing node access checks. The node access data stored in the index is based on the node_access table which is affected by hook_node_access_records(). The data alteration is only available for node indexes.
Search views do not filter based on node access by default. There is a simple option in the query settings called "Additional access checks on result entities" that will do an access check after the actual query is run, but this option should only be used as a last result. Search results counts and facets will not reflect the further restriction applied by views.
The proper way to do the node access checks in views is to add a filter on Indexed Node: Node access information. This can be complicated because it is important to know what values the field will hold, and this information can not be output through fields in the view itself. One must look to the data stored in the search server. In the case of Solr, this can be accomplished by examining the sm_search_api_access_node field in the schema browser. A sample value for one configuration of taxonomy access control was node_access_taxonomy_access_role:2. One could make a views search display for authenticated users for example that included all results, and a display for anonymous users that checked that the Node Access Information value is not equal to node_access_taxonomy_access_role:2 in the example above. A brief example can be found in this Drupal Answers answer
URL field Adds a field containing the URL at which the entity can be displayed. For some item types, like nodes, this URL is already available, but this data alteration can be used to also add them for other types.
Aggregated fields Offers the ability to add additional fields to the entity, containing the data from one or more other fields. Use this, e.g., to have a single field containing all data that should be searchable, or to make the text from a string field, like a taxonomy term, also fulltext-searchable. The type of aggregation can be selected from a set of values: you can, e.g., collect the text data of all contained fields, or add them up, count their values, etc.
Complete entity view Adds a field containing the whole HTML content of the entity as it is viewed on the site. The view mode used can be selected. This allows you to index exactly „what the user sees“, which is often what is expected, but might differ from just indexing the contents of other fields. Note that this might not work for items of all types. All core entity types except files are supported, though.
Index hierarchy Allows you to index hierarchical fields along with all their parents. Most importantly, this can be used to index taxonomy term references along with all parent terms. This way, when an item, e.g., has the term New York, it will also be matched when filtering for USA or North America.
- Processors
A processor can do preprocess data that is being indexed, preprocess search queries, and postprocess search results Refer to service class documentation of the server.
Ignore case Makes searches on selected fields case-insensitive. Some servers might do this automatically, for all others this should probably always be activated, at least for fulltext fields.
HTML filter Strips HTML tags from selected fields and decodes HTML entities. If you are indexing HTML content (like node bodies) and the search server doesn't handle HTML on its own, this should be activated to avoid indexing HTML tags, as well as to give e.g. terms appearing in a heading a higher boost.
Tokenizer This processor allows you to specify how indexed fulltext content is split into seperate tokens – which characters are ignored and which treated as white-space that seperates words.
Stopwords Enables the admin to specify a stopwords file, the words contained in which will be filtered out of the text data indexed. This can be used to exclude too common words from indexing, for servers not supporting this natively.
Highlighting Adds highlighting of search terms to the search results.
- Data alterations
2.4.49.3. Create a search view or a search page
https://www.drupal.org/docs/7/modules/search-api/getting-started/search-forms-and-results-pages/search-api-views Admin > Structure > Views > Add new view (admin/structure/views/add)
View name: Search Show: [the name of the Search index] (in the case of fuzzysearch the default is "Default fuzzysearch index") Create a page [tick] Page title: Search Path: search Display format: Unformatted list of Rendered entity Items to display: 10, use pager Continue & edit (the new View)
Format: Show: Rendered entity | Settings
View mode: Search results Filter criteria
Fulltext search: Expose this filter, Required, Remember the last selection, Use as: search keys Sort criteria
Search: Relevance, descending (if you don’t have an order with fuzzysearch you will get a PDO exception) Page settings
Access: Permission: view published content Advanced
No results behaviour: Global: Text area “No results matched your search.” Exposed form
Exposed form in block: Yes (This option will only show up on Page displays.) Exposed form style settings: Submit button text: Search Save the View. Add the exposed form block to a region. Note that you will only receive results for partial matches that are longer than the minimum word length specified in the Index configuration.
Views based on Search API use as base table a search_api_index_* table, not the usual node table or other ones supported by Views. That is why the Search views are selected at the views' creation moment, and it can not be changed later.
2.4.49.4. Convert an existing view to search api
If the views to be converted use the row plugins Content (node), Fields or Rendered entity, the conversion may be easier (see further). If not (the selected row plugins were others), then unfortunately the views should be fully recreated manually: create a brand new search_api view and recreate all the elements from your old view by hand.
For views using Content, Fields or Rendered entity row plugins: Export the view to be converted. Save this code as a backup. Modify the view, removing all the Filters (contextual or not), all the Sort criteria and all the Relationships. Write down the removed items: you will need to recreate them manually using the indexed versions. When the view is clean of filters and sort criteria, export it again. In the export code, the 5th line will contain the $view->base_table variable. Change it to the search api index to be used in the view (the index must have been created before):
$view->base_table = 'search_api_index_<name_of_your_index>'; Maybe fiddle around with fields (see below). Go to admin/structure/views/import and import the modified code. Do not forget to select the option Replace an existing view if one exists with the same name. (NOTE: currently there is a bug in Ctools 1.x that makes the Import option not to be shown for administrator users different than uid=1, here is the issue: #870938: Add new permission for controlling imports ). Add the previously removed Filters, Sort criteria and Relationships, selecting them from the now available indexed versions. Process will be easy if the views to be converted used the Rendered entity row plugin, since no changes will be required for the row contents.
Field fiddling
For fields there is no general recipe, although these hints should catch most cases:
Care that your index contains all needed fields or create them when import errors bug you.
Look for lines that look like this:
$handler->display->display_options['fields']['field_FOO']['table'] = 'TABLE'
If TABLE is 'views', don't change.
If TABLE is 'node' (or your base table) or 'field_data_FOO', change to 'search_api_index_<name_of_your_index>'
2.4.49.5. Configure Facet, Facet Blocks
A block is created for each facet configured. Even though it's set to appear on all pages, it won't display for a search view or a search page.
2.4.49.6. Views Filter > Search: Fulltext search
2.4.49.7. Current Search Block
current_search is installed with d7:facetapi. /admin/config/search/current_search
2.4.49.8. Other useful modules
2.4.49.9. Developer Documentation
https://www.drupal.org/docs/7/modules/search-api/developer-documentation https://www.drupal.org/docs/7/modules/search-api/advanced-site-building-tutorials
- Execute a search in code
// Replace "node_index" with you index's machine name. $query = search_api_query('node_index'); $query->condition('type', 'product', '='); $filter = $query->createFilter('OR'); $filter->condition('field_category', 'book', '='); $filter->condition('field_category', 'magazine', '='); $query->filter($filter); $data=$query->execute(); $results=$data['results']; print_r($results);
- Combined fields to do IN or OR across multiple fields
https://www.drupal.org/docs/7/modules/search-api/advanced-site-building-tutorials/combining-fields https://www.drupal.org/project/search_api_combined
Limitation At present the module has been tested for multiple term references and not with any other field type. It is currently set to index all combined fields as lists of items so it will not work with multiple fields that have single values that expect to stay as single values – where this is important is in sorting, for instance, since you cannot sort by multiple values with Solr.
2.4.49.10. API of Search API
http://www.drupalcontrib.org/api/drupal/contributions%21search_api%21search_api.api.php/7 path-to-mod/search_api/search_api.api.php
Search search_api*.api.php search_api_solr.api.php
2.4.50. Search API Solr - search_api_solr - Package: Search
search_api_solr requires d7:search_api and core Search d7:search_api_solr It's more powerful than module apachesolr It creates View Mode called Search Index in each content type
Setup an account and create an index on Opensolr.com
On OpenSolr UI, upload the 4.x or corresponding Solr version based on OpenSolr version. path-to-mod/search_api_solr/solr-conf/4.x https://opensolr.com/blog/2011/09/how-to-use-with-drupal
Lucene version :: search lucene in solr-conf/4.x, e.g. solr.luceneMatchVersion=LUCENE_40 <luceneMatchVersion>${solr.luceneMatchVersion:LUCENE_40}</luceneMatchVersion>
In bundle (e.g. node type abc), setup Manage Display for the Search Index.
2.4.50.1. Debug
In OpenSolr UI for an index (tnt_dev), there's a button Browse Data. https://path-to-aws.opensolr.com/solr/tnt_dev/select?q=*:*&wt=json&indent=true&start=0&rows=5
Insert a key-value pair to search https://path-to-aws.opensolr.com/solr/tnt_dev/select?q=im_field_industry:337&wt=json&indent=true&start=0&rows=5
Retrieve the request made to Solr and response from Solr
In search_api_solr/includes/solr_connection.inc add a line below protected function makeHttpRequest
drupal_set_message(check_plain($url)); // <-- Add this line to retrieve request $result = drupal_http_request($url, $options); drupal_set_message(check_plain($result->data)); // <-- Add this line to retrieve response
2.4.51. Search API Solr Overrides - search_api_solr_overrides
Requires d7:search_api_solr Provides site/environment specific overrides for search_api_solr configuration in settings.php
2.4.52. Views
2.4.53. Better Exposed Filters
better_exposed_filters d7:better exposed filters Requires d7:ctools d7:views Group under Views 7.x-3.3, 7.x-3.4
2.4.54. Views Bulk Operations
views_bulk_operations : sub module actions_permissions d7:views_bulk_operations Requires: d7:entity d7:views
2.4.55. Administration views
2.4.56. Media
media Requires: d7:file entity, d7:ctools, d7:views
From 7.x-2.0-beta1 to 7.x-2.9
2.4.57. Internationalization - i18n
i18n requires core locale and d7:m:variable
- Add a language
- admin/config/regional/language
- (no term)
- Go to pages that use t('string to translate', [], ['langcode'=>"fr"]) with non default options
Once a language is added (default language is chosen), node created will have $node->language = 'en'
2.4.58. S3 File System - s3fs d7:m:s3fs
- https://pantheon.io/docs/drupal-s3/
- Requires module libraries, library
AWS SDK for PHP 2.x, PHP 5.3.3+ withallow_url_fopen = Oninphp.ini - It replaces
public://orprivate://withs3:// - After this is enabled, files that are in Drupal but not in S3 will become unavailable! Drupal files need to be uploaded to S3 first!
- Without replacing
public://withs3://, only new files will be uploaded to S3. If a node has files inpublic://and they are not uploaded tos3://andwithout replacing public://is set, saving the node will not change the files frompublic://tos3://. In short, it's safe to not enablereplace public:// with s3:// drush s3fs-copy-local. Refer to module README.txt for more info- In order to sync files from S3 to Drupal, Refresh file metadata cache is needed
/admin/config/media/s3fs/actions. Best to run drush in case of timeoutdrush s3fs-refresh-cache
drush dl s3fs drush en s3fs # update.php is not necessary # Download library # In Pantheon, drush make --no-core code/sites/all/modules/s3fs/s3fs.make code # Or drush make --no-core sites/all/modules/s3fs/s3fs.make
- Check if library is downloaded
sites/all/libraries/awssdk2/aws-autoloader.php - /admin/config/media/s3fs/settings or in settings.php
- Set
Cache-Controlheader to bepublic, max-age=604800 - Refer to header:cache-control
settings.php
$conf['awssdk2_access_key'] = 'YOUR ACCESS KEY'; $conf['awssdk2_secret_key'] = 'YOUR SECRET KEY'; $conf['s3fs_root_folder'] = 'thenyourroot'; // bucketroot/thenyourroot/* $conf['s3fs_bucket'] = 'YOUR BUCKET NAME'; $conf['s3fs_region'] = 'YOUR REGION'';
- Set
Default download methodtoAmazon Simple Storage Service - Go to
admin/config/media/file-system - Set
Upload destinationfor a file field for a content type - Strucutre > Content Types > choose one > choose the file field
- (no term)
- Refer to aws:s3 about how to setup user with correct permission policy
2.4.58.1. IMCE & S3FS, TinyMCE
If S3fs is < 7.x-2.5 then get the latest s3fs dev version.
Any field of type File, Image, etc. can set the "Upload destination" to S3 in the Field Settings.
To insert javascript inside Wysiwyg with TinyMCE (v3.5.8), first in admin allow user to turn off TinyMCE by default.
/admin/config/content/wysiwyg/profile/full_html/edit, Basic Setup, Allow users to choose default
Login to the user profile, and turn off Text formats enabled for rich-text editing > Full HTML
That user can still enable rich-text but the default is plain text without stripping script tags.
However, toggle rich-text on and off still strips off script tags.
In short, it's hacky to enable users to insert javascript. You may need the shortcode module.
2.4.59. Password Policy - password_policy
Change password required length, etc.
2.4.60. Secure Login - securelogin
It enforces secure authenticated session cookies and ensures the user login page, any pages with the user login block and other forms you configure are submitted securely via HTTPS.
2.4.61. Secure Review - security_review
Automate testing
2.4.62. Security Kit - seckit
- https://www.drupal.org/project/seckit
- Refer to header:csp
- Content Security Policy (CSP) implementation via HTTP response header
Content-Security-Policy - Control over Internet Explorer / Apple Safari / Google Chrome internal XSS filter via
X-XSS-ProtectionHTTP response header - Prevent content upsniffing and serving files with incorrect MIME-type via
X-Content-Type-Options: nosniffHTTP response header
2.5. Debug
Enable module Database Logging and Syslog if necessary.
watchdog('LOG_TYPE','message'); watchdog('LOG_TYPE', $message, array(), WATCHDOG_ERROR); array() is to t() $message. Don't forget to put empty array otherwise you will have to manually clean up the watchdog table!
If error or warning messages are printed using drupal_set_message, modify the `drupal_set_message` function so that it prints debug_backtrace
if ($type == 'error') { $message .= ' '. print_r(debug_backtrace(DEBUG_BACKTRACE_PROVIDE_OBJECT,3),1); }
You can throw an Exception
throw new Exception(print_r($val));
if (ip_address() === '123.456.789.123') { var_dump(); } // can be used to see what files include this file var_export(debug_backtrace()); // Just return the last called function $dbt = debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS,2); $caller = isset($dbt[1]['function']) ? $dbt[1]['function'] : null;
2.5.1. debug_backtrace
- $options
- DEBUG_BACKTRACE_PROVIDE_OBJECT
- default. Include everything
- DEBUG_BACKTRACE_IGNORE_ARGS
- Don't include function/method arguments
- index 0 being the most recently run code, the last index is the entrypoint
- Can be used to see which parent file includes a file
- If the current calling place is the first script file that is run, it will return an empty array
debug_backtrace(int $options = DEBUG_BACKTRACE_PROVIDE_OBJECT, int $limit = 0): array
2.6. Configuration, Setting, Maintenance Mode
Override configuration and setting based on the current dev environment settings.php
$conf['admin_theme'] = 'seven'; $conf['maintenance_mode'] = FALSE;
- To determine which environment
- pantheon:environment
When site is under maintenance and you want to login:
- Turn off javascript in browser
- a.com/?q=user
UPDATE `variable` SET `value`='0' WHERE `name` = 'maintenance_mode'
For assigning permissions to roles, e.g. enable anonymous for Devel, you have to do it manually
Variable name and its default value
- admin_theme
- 'seven'
- preprocess_css
- "1"/"0", aggregate.
- preprocess_js
- "1"/"0", aggregate.
- cache
- "1"/"0", cache pages for anonymous users
- block_cache
- "1"/"0", cache blocks
- cache_lifetime
- "0", none. Minimum cache lifetime
- error_level
- "0""1""2", none/error and warnings/all messages. Logging and errors.
If setting is an array, you have to override the whole array. Can't override a key value.
2.6.1. d7:mysql:charset
- Since Drupal 7.50, Drupal now supports 4 byte UTF-8 with MySQL. You will first need to run a custom drush command provided by a utf8mb4_convert module to convert all Drupal database, tables and fields to charset
utf8mb4and collationutf8mb4_general_ci. Then change the database connection settings in settings.php - Drupal Doc
- The utf8mb4_convert module requires
inno_large_prefix=truewhen MySQL server is bootup, Otherwise a test->utf8mb4IsSupportedcan't pass before any actionsTo bypass the test, re-create any string index columns
varchar(255)tovarchar(191)ALTER TABLE cache_block CHANGE cid cid VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
- Do this for any problematic column the module returns
- For new website, changing the database connection settings in settings.php is good enough
- If MySQL server doesn't bootup with
inno_large_prefix=true, all indexed columns will have maxvarchar(191) - As a quick workaround, you can remove all php:4byte characters
- Refer to mysql:charset for important info
2.6.2. Show error d7:show error
See 2.5
error_reporting(E_ALL); ini_set('display_errors', TRUE); ini_set('display_startup_errors', TRUE); $conf['error_level'] = 2; // UI > Administration > Configuration > Development > logging > 'All messages'
2.6.3. Session Timeout, Session Cookie Timeout
// these 2 lines are to make sure PHP has the settings to enable gc ini_set('session.gc_probability', 1); ini_set('session.gc_divisor', 100); // about 2 days for session timeout ini_set('session.gc_maxlifetime', 200000); // about 20 days for session cookie timeout ini_set('session.cookie_lifetime', 2000000);
2.7. URL Alias
- Setup pattern for each node type
- Configuration > Search and metadata > URL Aliases > Patterns
2.8. Link, Module Path, Theme Path, File Path
2.8.1. Link
Empty link!
// Basic syntax: l($text, $path, array $options = array()) l($text, '', array('fragment'=>' ', 'html'=>TRUE, // if $text is HTML, set this to true 'external'=>TRUE, // 'attributes' => array( // 'target'=> '_blank' // 'class' => ['classA'], // ), ) );
- Function l add
class="active"when the path is current_path(). Use url() instead.
url($path = NULL, $options = array());
- return
- string of URL
- $path
- internal or external e.g.
node/34http://example.com/foohttp://a.com/?foo=bar - $options
- append components
- query
- an array of query key/value-pairs (without any URL-encoding) to append to the URL. Result is URL encoded
- fragment
- A fragment identifier (named anchor) to append to the URL. Do not include the leading '#' character.
url($path = NULL, $options = array())
Get URL alias url() just returns a url. $options are the same as l(). url('taxonomy/term/1');
- Embed a url directly to HTML href
check_url($url)- Embed a url directly as a URL parameter
echo 'http://abc.com/?t='.urlencode($url);
2.8.2. Current External URL, d7:url d7:current_path
url(current_path(), array('absolute' => TRUE));
;; With query parameters
url(current_path(), array('absolute' => TRUE, 'query' => drupal_get_query_parameters()));
current_path still works when 404. Don't var_dump or print current_path() directly as it might have XSS
url
- absolute
- append http
- base_url
- override the default $base_url with a custom base url, e.g. http://abc.com without trailing stash
2.8.3. Module Path, Theme Path, File Path
global $base_url; $mod_path = $base_url.base_path().drupal_get_path('module', 'lili_mod_name'); $theme_pat = $base_url.base_path().drupal_get_path('theme', 'lili_theme_name'); // web root path echo filemtime(getcwd().'\sites\all\themes\express\css\styles.css');
Both have no trailing slash
2.8.4. global $base_url
No trailing slash http://abc.com
2.9. Translate
2.9.1. t()
$text = t("This is !name's website", array('!name' > $username));
$text = t("This is @name's website", array('@name' => $username));
$text = t("This is %name's website", array('%name' => $username));
$text = t(check_plain($text));
$link = l(t("Link text"), "node/123");
$link = t('Visit the <a href"@url">settings</a> page', array('@url' => url('admin')));
/ `%` and `@` use `check_plain()` / `%` adds `<em class="placehoder"></em>` // `!` straight output.
t('Thursday',[],['langcode'=>'fr']); t('Thursday',[],['context'=> 'some context', 'langcode'=>'fr']);
2.9.2. Translate Date and Time
// Have to load all terms then Translate Interface can show up admin/config/regional/translate/translate
$_fr_days_translation = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'];
$_fr_months_translation = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];
foreach ($_fr_days_translation as $_t_i) {
t($_t_i, [],['langcode'=>'fr']);
}
// format_date with F (long month name) adds context..
foreach ($_fr_months_translation as $_t_i) {
t($_t_i, [],['context'=>'Long month name','langcode'=>'fr']);
}
$_u_time = mktime (11, 0, 0, $_u_time['month'], $_u_time['day'], $_u_time['year']);
$_fr_time = format_date($_u_time, 'custom', 'l j F Y', NULL, 'fr');
echo $_fr_time;
2.10. Global Variables
- DRUPAL_ROOT
/index.php- VERSION
- 7.67
- (no term)
- https://api.drupal.org/api/drupal/globals/7.x
- $base_url
- 'http://www.yoursite.com' wihtout trailing slash
- $base_path
- $base_root
- $databases
- $cookie_domain
- $conf
- $installed_profile
- $upodate_free_access
- $db_url
- $db_prefix
- $drupal_hash_salt
- $is_https
- $base_secure_url
- $base_insecure_url
- (no term)
- Drupal Properties
$_SERVER['*']- QUERY_STRING
- query string without q e.g. 'foo=bar&koo=hoo'
2.11. Common Functions d7:functions
2.11.1. drupal_add_html_head
drupal_add_html_head($tag, 'unique-name'); // $head in html.tpl.php $tag = array( '#tag' => 'meta', '#attributes' => array( 'property' => 'og:url', 'content' => $base_url, ), ); drupal_add_html_head( $meta_og, 'lili_og_url' );
2.11.2. drupal_set_title
drupal_set_title($title = NULL, $output = CHECK_PLAIN);
- $title
- if NULL, leave the current unchanged
2.11.3. drupal_goto
drupal_goto($path = '', array $options= array(), $http_response_code= 302)
- $path
- a drupal path 'node/123' or a full url
- $options
- URL optoins to pass to url()
Redirect to homepage drupal_goto('<front>');
2.11.4. drupal_valid_path($path, $dynamic_allow= TRUE)
$_path = current_path();
if (drupal_valid_path($_path)) { // Current user has access to this path/page }
2.11.5. drupal_clean_css_identifier
Convert string to CSS ready element name, class and ID in selectors.
echo drupal_clean_css_identifier(drupal_strtolower($class));
2.11.6. Sanitization functions d7:functions:sanitization
https://api.drupal.org/api/drupal/includes%21common.inc/group/sanitization/7.x
check_plain($text)- refer to PHP htmlspecialchars. Can be used to sanitize HTML attribute value. d7:functions:check_plain
- drupal_strip_dangerous_protocols($uri)
- not for display.
drupal_attributes(array $attributes = [])- Each array key and its value will be formatted into an attribute string. Attr. values are check_plain() but not the key (attribute name).
- check_url($uri)
- strip and encode a uri for output to HTML. eq. to check_plain(drupal_strip_dangerous_protocols)
- filter_xss($string, $allowed_tags = ['a', 'em', …])
- Filtes HTML to prevent XSS vulnerabilities.
2.12. Theme API
2.12.1. render() vs drupal_render()
render(&$element) render is a wrapper for durpal_render which
- makes sure the element passed in is set to be shown show()
- only renders array. If what's passed in is not an array, it will return as it is.
show() :: $element['#printed'] = FALSE;
drupal_render(&$elements)
- if #access == false, return ''
- if #printed = true, return ''
- if #cached is set, load and return cache drupal_render_cache_get($elements)
- #markup is set but not #type, #type is set to markup
- array of pre_render function names to modify
- pre_render functions might set #printed to some value to indicate not to print, if so return ''
- get children $children and initialize #children
- if #theme is set, theme all children and save to #children
- if #children is still empty, concatenate drupal_render for each child
- if #theme_wrappers is defined, run each theme_wrapper function for #children
- if #post_render is defined, run each function for #children
- if #states is defined, attach JavaScript state drupal_process_states
- if #attached is defined, drupal_process_attached
- #prefix . #children . #suffix
- drupal_render_cache_set
- #printed set to true
2.12.2. theme()
- https://api.drupal.org/api/drupal/includes%21theme.inc/function/theme/7.x
theme($hook, $variables = array())- Use
$variablesto merge with defaults from the theme registry
2.12.3. System Theme
- Default theme files
*.tpl.phpare located at/modules/mod_name/mod-name-component-name.tpl.php - e.g.
/modules/node/node.tpl.php - For custom theme, better to put
*.tpl.phpfiles under/sites/all/themes/theme_name/templates/
2.12.3.1. Fast detect if it's a node page
if (arg(0)=='node') { $node_id = arg(1); }
2.12.3.2. Include file in *.tpl.php
include(drupal_get_path('theme', 'theme_name').'/example-file-template.tpl.php');
If the file is in the same folder <?php include 'abc.php'; ?>
2.12.3.3. .info (required)
2.12.3.4. html.tpl.php
Get node if it's a node page $node = menu_get_object(); if (isset($node->type)) { … }
2.12.3.5. page.tpl.php
Get node if it's a node page hook_preprocess_page(&$variables) { if (!empty($variables['node']) && $variables['node']->type == 'NODETYPE') { $variables['greeting'] = 'Custom Greeting'; } }
2.12.3.6. region.tpl.php
2.12.3.7. block.tpl.php
2.12.3.8. node.tpl.php d7:template:node
- Available variables
- sanitized title of the node
- $content['field_example']
$node $type :: node type
Node status variables $view_mode :: 'full'
- Check if a field is empty
- $content['field_name']
print render($content['field_name']) is based on the view mode that the entity is currently used to display and then the field formatter.
If field_name is not a field of the $node, then $content['field_name] is not set. If field_name is defined but it has no value, then $content['field_name'] is not set. So you can't directly render($content['field_name']) because the field_name might not be defined or has value.
render($content['field_name']); //Raw value $delta = 0; // first value. Even for text field $item = $content['field_name']['#items'][$delta] $item['target-id'] // for entity reference field $item['entity']->title // entity title $aField = $item['entity']->field_entity_field_name // array $aField['und'][0]['value'] // raw value of a field of that entity $aField['und'][0]['safe_value']
2.12.3.9. field.tpl.php
2.12.3.10. comment-wrapper.tpl.php
2.12.3.11. comment.tpl.php
2.12.3.12. template.php
2.12.3.13. logo.png
2.12.3.14. screenshot.png
2.12.4. Form Theme
$form['#theme'][] = 'lili_form'; If no custom theme function, d7:hook_theme d7:theme_*, is defined, then the default template file is /sites/all/themes/your_theme/templates/form/lili_form.tpl.php
2.12.5. Region
Region can hold multiple blocks. If module d7:blockgroup is installed, a block group is a region but it can be included in other region or block group.
Define a region in mytheme.info e.g. regions[header] = Header
Render a region in template files print render($page['header'])
2.12.6. Block
2.12.6.1. hook_block_info
function hook_block_info() {
$blocks = array();
$blocks['my_block'] = array(
'info' => t( 'My Custom Block' ),
'cache' => DRUPAL_CACHE_PER_ROLE,
);
return $blocks;
}
Use `DRUPAL_NO_CACHE` for development, then `DRUPAL_CACHE_GLOBAL`. Default is `DRUPAL_CACHE_PER_ROLE`
If `Cache Block` is disabled and unchecked in Drupal Performance setting, all blocks will not be cached. Refer to d7:cache block
The delta `my_block` only needs to be unique within the module. HTML ID is `block-MODULENAME-DELTA`
Now the block can be seen in the default theme Blocks admin page /admin/structure/block Place the new block to a region. To assign a block to multiple regions, install d7:MultiBlock module: Then go /admin/structure/block/instances to create another instance of the newly created block type.
2.12.6.2. hook_block_view
<?php
/**
* Implements hook_block_view().
*/
function lili_block_view( $delta = '' ) {
$block = array();
switch ( $delta ) {
case 'my_block' :
$block['content'] = _lili_my_block_view();
break;
}
return $block;
}
/**
* Custom function to assemble renderable array for block content.
* Returns a renderable array with the block content.
* @return
* returns a renderable array of block content.
*/
function _lili_my_block_view() {
$block = array();
// Capture the image file path and form into HTML with attributes
$image_file = file_load( variable_get( 'block_image_fid', '' ) );
$image_path = '';
if ( isset( $image_file->uri ) ) {
$image_path = $image_file->uri;
}
$image = theme_image( array(
'path' => ( $image_path ),
'alt' => t( 'Image description here.' ),
'title' => t( 'This is our block image.' ),
'attributes' => array( 'class' => 'class_name' ),
) );
// Capture WYSIWYG text from the variable
$text = variable_get( 'text_variable', '' );
// Block output in HTML with div wrapper
$block = array(
'image' => array(
'#prefix' => '',
'#type' => 'markup',
'#markup' => $image,
),
'message' => array(
'#type' => 'markup',
'#markup' => $text,
'#suffix' => '',
),
);
// directly return a form as block
// return drupal_get_form('lili_another_form');
// or return ['some thing' => drupal_get_form('lili_another_form')];
return $block;
}
2.12.6.3. hook_block_view_alter
function lili_block_view_alter( &$data, $block ) {
if ( $block->module == 'block' && $block->delta == '3' ) {
// $data['subject']: title of the block
$data['content'] = '';
}
}
2.12.6.4. Configurable Block: hook_block_configure, hook_block_save
hook_block_configure
/**
* Implements hook_block_configure().
*/
function lili_block_configure( $delta = '' ) {
$form = array();
switch ( $delta ) {
case 'my_block' :
// Text field form element
$form['text_body'] = array(
'#type' => 'text_format',
'#title' => t( 'Enter your text here in WYSIWYG format' ),
'#default_value' => variable_get( 'text_variable', '' ),
);
// File selection form element
$form['file'] = array(
'#name' => 'block_image',
'#type' => 'managed_file',
'#title' => t( 'Choose an Image File' ),
'#description' => t( 'Select an Image for the custom block. Only *.gif, *.png, *.jpg, and *.jpeg images allowed.' ),
'#default_value' => variable_get( 'block_image_fid', '' ),
'#upload_location' => 'public://block_image/',
'#upload_validators' => array(
'file_validate_extensions' => array( 'gif png jpg jpeg' ),
),
);
break;
}
return $form;
}
/**
* Implements hook_block_save().
*/
function lili_block_save( $delta = '', $edit = array() ) {
switch ( $delta ) {
case 'my_block' :
// Saving the WYSIWYG text
variable_set( 'text_variable', $edit['text_body']['value'] );
// Saving the file, setting it to a permanent state, setting a FID variable
$file = file_load( $edit['file'] );
$file->status = FILE_STATUS_PERMANENT;
file_save( $file );
$block = block_load( 'lili', $delta );
file_usage_add( $file, 'lili', 'block', $block->bid );
variable_set( 'block_image_fid', $file->fid );
break;
}
}
2.12.6.5. Block API Sequence
hook_block_configure: Define a configuration form for a block. hook_block_info: Define all blocks provided by the module. hook_block_info_alter: Change block definition before saving to the database. hook_block_save: Save the configuration options from hook_block_configure(). block_list
- _block_load_blocks
- hook_block_list_alter
- _block_render_blocks: render a set of blocks for one region. Check if it's cacheable, then get and set cache_block
- _block_get_cache_id
- hook_block_view: Return a rendered or renderable view of a block.
- hook_block_view_alter: Perform alterations to the content of a block.
- hook_block_view_MODULE_DELTA_alter: Perform alterations to a specific block.
2.12.6.6. Render a block
If a block is assigned to a region "content" Markup of the the block is $page['content']['block_123']
$SIDEBAR_CONTENT = json_encode($page['content']['block_'.$block_id]); $SIDEBAR_CONTENT = json_decode($SIDEBAR_CONTENT,true); $SIDEBAR_CONTENT = $SIDEBAR_CONTENT['#markup'];
2.12.7. Custom Theme Function
Module theme: theme('modname_theme-name',$data)
d7:hook_theme
- $existing
- array. existing themes
- $type
- string. Whether a theme, module, etc. is being processed. e.g. Is it a parent theme?
- $theme
- the name of theme, module, etc. is being processed
- $path
- the directory path of theme or module
Return an array
- variables
- theme function takes no parameter array(). Assign null to key-value for default values.
- path
- Use with 'template'
- If theme_* function is used, you don't need to specify 'path' and 'template'
- Whether or not hook_theme is defined in template.php or in a module, the path default is to the module or theme
- But you should make the path to sites/all/themes/your_theme/templates/ or sites/all/modules/mod_path/modname/templates
- e.g. 'path' => drupal_get_path('module', 'panels') . '/templates', 'path' => drupal_get_path('theme', 'panels') . '/templates',
- if path is not defined but template is defined as a path to file (theme/template_file_name), then
- hook_theme defined in template.php
- path_to_theme/theme/template_file_name.tpl.php
- hook_theme defined in a module
- path_to_module/theme/template_file_name.tpl.php
- if path is not defined but template is defined as a file (template_file_name), then
- hook_theme defined in template.php
- path_to_theme/template_file_name.tpl.php
- hook_theme defined in a module
- path_to_module/template_file_name.tpl.php
- template
- Assuming the theme function name is theme_lili_flair
- Default file name is lili_flair and the actual file is lili_flair.tpl.php
function lili_theme( $existing, $type, $theme, $path ) { return array( 'lili_flair' => array( 'variables' => array( 'text' => NULL, ), ), ); } function theme_lili_flair( $variables ) { if ( ! empty( $variables['text'] ) ) { return '<span class="li-tag li-tag-pill li-tag-color">' . $variables['text'] . '</span>'; } else { return ''; } } echo theme('lili_flair',array('text'=>'hi')); // In xxx.tpl.php use $text directly
2.12.8. Sub theme
- https://www.drupal.org/docs/7/theming/creating-a-sub-theme
- The name of your sub-theme must start with an alphabetic character and can only contain lowercase letters, numbers and underscores
- Folder
my_subtheme- .info file
name = My Subtheme
base theme = theme_name- The sub-theme inherits most properties of the base theme. The important exceptions are
- regions
- core version
- color info
- features
- parent theme's logo (logo.png/logo.jpg)
- parent theme's favicon.ico
- theme-settings.php
- May want to copy the regions section of your base theme's info file, along with its core version declaration
If your base theme supports the color module and you'd like your sub-theme to support it, you probably also want to copy the color folder from your base theme and add the line from your base theme's info file to your sub-themes info file that looks like:
stylesheets[all][] = css/colors.css
and then copy the colors.css from base theme to the css folder of the sub-theme.
Style sheets and JavaScripts are inherited. If you want to override, create relevant files and add these to .info
stylesheets[all][] = style.css scripts[] = script.js features[] = logo
template.phpfunciton inheritance- Anything defined in the parent theme's template.php file will be inherited. This includes theme function overrides, preprocess functions and anything else in that file
- Each sub-theme should also have its own template.php file, where you can add additional functions or override functions from the parent theme
- theme functions, e.g.
theme('[hook]', $var,...), When a sub-theme overrides a theme function, no other version of that is called - preprocess functions, e.g.
[theme]_preprocess_pageis called before page.tpl.php is rendered- Unlike theme functions, preprocess functions are not overriden in a sub-theme
- Instead, the parent theme preprocess function will be called first, and the sub-theme preprocess function will be called next
- There is no way to prevent all functions in the parent theme from being inherited. The only way to remove a parent themes' preprocess function is through
hook_theme_registry_alter()
- Page, node, block and other template (.tpl.php) file inheritance
node--blog.tpl.phpbuilding on an inheritednode.tpl.php- A single hyphen is still used to separate words. The double hyphen always indicates a more targeted override of what comes before the
--. e.g.node--long-content-type-name.tpl.php
- Add a template file with the same name in sub-theme folder to have it override the template from the parent theme
- Screen shots inheritance
name = Jean description = A subtheme of Bartik, which is a flexible, recolorable theme with many regions. core = 7.x base theme = bartik ; Add a style sheet for all media stylesheets[all][] = css/jean.css stylesheets[all][] = css/colors.css ; Add a style sheet for screen and projection media ; stylesheets[screen, projection][] = theScreenProjectionStyle.css ; Add a style sheet for print media ; stylesheets[print][] = thePrintStyle.css ; Add a style sheet with media query ; stylesheets[screen and (max-width: 600px)][] = theStyle600.css ; avoid using *_style.css but *style.css e.g. myownstyle.css is ok.. regions[header] = Header regions[help] = Help regions[page_top] = Page top regions[page_bottom] = Page bottom regions[highlighted] = Highlighted regions[featured] = Featured regions[content] = Content regions[sidebar_first] = Sidebar first regions[sidebar_second] = Sidebar second regions[triptych_first] = Triptych first regions[triptych_middle] = Triptych middle regions[triptych_last] = Triptych last regions[footer_firstcolumn] = Footer first column regions[footer_secondcolumn] = Footer second column regions[footer_thirdcolumn] = Footer third column regions[footer_fourthcolumn] = Footer fourth column regions[footer] = Footer settings[shortcut_module_link] = 0
- General steps to create a sub theme
- Create the folder
/sites/all/themes/jean- Create
.infofile /sites/all/themes/jean/jean.info- Create a blank file named
/sites/all/themes/jean/css/jean.css- Copy from bartik or create your own
/sites/all/themes/jean/logo.png- (no term)
- In order to get the color module to work with your subtheme, you will need to do the following:
- Copy the file
/themes/bartik/css/colors.cssto/sites/all/themes/jean/css/colors.css - Copy the folder and its contents
/themes/bartik/color/to/sites/all/themes/jean/color/
- Copy the file
- (no term)
- Go to the Administration > Appearance page to enable your new subtheme called Jean. Now you can add CSS to your jean.css file, and it will apply to your new subtheme
2.12.9. 404 page
page–404.tpl.php could be in theme folder or theme's templates folder
function jean_preprocess_page(&$vars) { $header = drupal_get_http_header('status'); if ($header == '404 Not Found') { $vars['theme_hook_suggestions'][] = 'page__404'; } }
2.12.10. Contributed themes
2.12.10.1. Omega
2.12.10.2. Corolla
2.12.11. Render a field of a node, a term
2.12.11.1. field_get_items, field_view_value, field_view_field
field_view_field returns a renderable array that has everything: lable and value. field_view_value returns a renderable array that only has the formatted value. Such as text field simply returns text
$delta = 0; // first value
$field = field_get_items('node', $node, 'field_youtube_link');
//
$output = field_view_value('node', $node, 'field_youtube_link', $field[$delta]);
// $output is an array
print render($output);
// it seams field_view_value can't render node title
$output
[ "#markup" => "...", "#access" => true ]
Taxonomy term field
$eventField = field_get_items('node', $node, 'field_event_category');
$eventTerm = field_view_value('node', $node, 'field_event_category', $eventField[0]);
$titleImLookingFor = $eventTerm['#title'];
Change value before rendering
$image = field_get_items('node', $node, 'field_image');
$output = field_view_value('node', $node, 'field_image', $image[0], array(
'type' => 'image',
'settings' => array(
'image_style' => 'thumbnail',
'image_link' => 'content',
),
));
2.12.11.2. Using node template
node--[node-type-machine-name].tpl.php- Refer to d7:template:node
In page.tpl.php
$node_view = node_view($node); // different view mode, default is 'full' // node_view($node, 'custom-view-mode'); // different language, default is the global content language of the current request // node_view($node, 'full', 'fr'); print drupal_render($test);
Add a view mode
/** * Implements hook_entity_info_alter(). */ function MODULE_entity_info_alter(&$entity_info) { $entity_info['node']['view modes']['block_feature'] = array( 'label' => t('Block feature'), 'custom settings' => TRUE, ); } /** * Implements hook_preprocess_node(). */ function MODULE_preprocess_node(&$variables) { if($variables['view_mode'] == 'block_feature') { $variables['theme_hook_suggestions'][] = 'node__' . $variables['type'] . '__block_feature'; } // add css files $node = $variables['node']; if (in_array($node->type, ['nodetype1','nodetype2','nodetype3'])) { drupal_add_css(drupal_get_path('theme', 'mytheme') . "/css/owl-carousel/owl.carousel.min.css"); drupal_add_css(drupal_get_path('theme', 'mytheme') . "/css/owl-carousel/owl.theme.default.min.css"); } }
Get raw value
echo $content['field_youtube_link']['#items']['0']['value'];
2.13. Drush
2.13.1. version drush version
2.13.2. Debug -vd
2.13.3. Alias
drush sa list all aliases
pantheon:drupal:alias
2.13.4. User
# one time login link. uid, user name, or email address for the user. Default is uid 1 drush user-login yourusername drush user-create newuser --mail="a@b.com" --password="password" drush user-add-role "administrator" newuser
2.13.5. Module
drush vset maintenance_mode 1 set to maintenance mode
drush vset maintenance_mode 0 remove maintenance mode
drush dl devel Download a module
drush en devel Enable a module
drush dis devel Disable a module
drush pmu devel Uninstall a module. Won't delete code files
drush up <modulename> -y Update a module. Don't need to visit update.php
drush up --no-core Update all modules.
drush pml List all modules that are enabled, disabled, not installed, installed.
drush pml --pipe list modules in code name
drush up <modulename> = drush upc <modulename> + drush updb
Remove incorrectly removed modules in db d7:ts:module is missing
2.13.6. Detect if it's drush
if (drupal_is_cli() && function_exists('drush_main')) return true;
return false;
2.13.7. Watchdog
2.13.7.1. Delete
;; delete all logs drush wd-del all drush wd-del --type=cron drush wd-del --severity=notice
2.13.7.2. Tail Recent Log Messages
drush watchdog-show --tail --full --count=50
2.13.8. SQL Query
drush sql-query "SELECT * FROM users WHERE uid=1"
2.13.9. Drupal Cache
Clear all Drupal Cache
drush cc all
2.13.10. Fields
// Field Name, Feild Type, Bundles drush field-info fields
// Field type, Default widget, Widgets drush field-info types
2.13.11. Custom Drush
hook_drush_command() and drush_hook_your_command() in hook.drush.inc
function li_drush_command() {
$items['feeds-import-tnt']=[
'description' => '...',
'options' => [
'feed-id' => ['description' => dt('Feed ID to import') ]
], // if there is no param, don't include 'options'
'examples' => [
'drush feeds-import-tnt --feed-id=123' => "Test example",
]
];
return $items;
}
function drush_li_feeds_import_tnt() {
$feed_id = drush_get_option('feed-id');
}
// drush feeds-import-tnt --feed-id=123
drupal_set_message()
- stdout. No logging. Can be used multiple times in a drush command.
[status]at the end of each line- not print out immediately
Use the following every 3 minutes to prevent drush command timeout
print "hello\n"; // print out immediately drush_print(); // drush_print uses print drush_print_r($array);
2.13.12. Pantheon Drush
2.14. Node API d7:api:node
https://api.drupal.org/api/drupal/modules%21node%21node.api.php/7.x https://api.drupal.org/api/drupal/modules%21node%21node.api.php/group/node_api_hooks/7.x
$node->nid :: node id $node->type
2.14.1. Update a node
$node = node_load($nid); $node->fieldname['und'][0]['value'] = 'field value'; node_save($node);
d7:field_attach_update doens't trigger hook_node_presave and other node hooks
$article_nodes = node_load_multiple(array(), array('type' => 'article'));
foreach ($article_nodes as $article_node) {
$article_node->body[LANGUAGE_NONE][0]['value'] = 'body value';
field_attach_update('node', $article_node);
}
Recommended: use d7:entity_metadata_wrapper to update to avoid putting language code
2.15. Views d7:viewsapi
2.15.1. UI
2.15.1.1. Contextual filter with term name not term id
Choose Content: Has taxonomy term ID
Check Specify validation criteria
Validator Taxonomy Term, Filter value type Term name converted to Term ID
Check Transform dashes in URL to spaces in term name filter values
2.15.1.2. Path vs Link
Content: Path with Rewrite Result option "Use absolute link" is the raw path.
2.15.1.3. Field value is taxonomy ID not taxonomy name
Use Rewrite Result "[field_name-tid]"
2.15.1.4. Row Class
Use token [field_my_field_name] in row class. For taxonomy term to show id, you will have to create another field [field_my_field_name_1] shown as plain text and don't create label and rewrite the field to show Raw id and then reference [field_my_field_name_1] in row class
2.15.1.5. Sort Tax Term Field
node has a field called Category which is Tax Term Cat123 In Views, add relationship Content:Category (the node field) and under SORT CRITERIA in View setting, add Taxonomy Term either Weight or Name and select the Taxonomy term Cat123.
2.15.2. Entity Reference View
Require d7:entityreference module d7:entity reference view
2.15.3. View Object
Hooks use $view object.
2.15.3.1. Properties
args :: array( 0 => '37', ) name :: 'tt_taxonomy' current_display :: 'page'
2.15.3.2. Methods
$v->get_items_per_page(); / Return nothong in hook_views_pre_build $v->set_items_per_page(10); / hook_views_pre_build
2.15.4. hook_views_api
Views 3.x
/**
* Implements hook_views_api().
*/
function my_module_views_api() {
return array(
'api' => 3,
);
}
All Views hook code should be placed in my_module.views.inc which is located in module root directory if 'path' is not defined. All hooks' callbacks (Class definition) such as plugin and custom filter callback should be included in my_module.info as separate files
2.15.5. hook_views_query_alter
Put it in MODULENAME.views.inc based on hook_views_api. But I had success to just include this hook in .module file One way to avoid setup the *.views.inc is to use Views GUI to assign an Query Tag then use hook_query_TAG_alter but it runs twice.. So stick with hook_views_query_alter
// dpm($query->where); // One condition $c1 = [ 'field' => 'node.status', 'value' => 1, 'operator' => '=' ]; $c2 = [ 'field' => 'node_search_index.word', 'value' => '% burger %', 'operator' => 'LIKE' ]; $c3 = [ 'field' => 'node.uid = :node_uid', 'value' => [':node_uid' => '5'], 'operator' => 'formula' ]; $c4 = [ 'field' => 'node.type', 'value' => ['ns_newsletter'], 'operator' => 'in' ]; // a condition group $cg1 = [ 'field' => object::Drupal\Core\Database\Query\Condition ]; [ 'conditions' => [ $c1, $c2 ] 'args' => [], 'type' => 'AND' ], [ 'conditions' => [ $c3, $c4 ] 'args' => [], 'type' => 'AND' ], [ 'conditions' => [ $cg1 ] 'args' => [], 'type' => 'AND' ]
function mymod_views_query_alter(&$view, &$query) { dpm($view, __FUNCTION__); dpm($query, __FUNCTION__); if ($view->name == 'lili_view' && $view->current_display == 'page') { foreach ($query->where as &$condition_group) { _recursively_alter_query_conditions($condition_group['conditions']); } } // add a where condition group // views_plugin_query_default::add_where($group, $field, $value = NULL, $operator = NULL) $cg_count = count($query->where); $query->add_where($cg_count, '0', '1', '='); } // helper function: (takes in conditions group argument) function _recursively_alter_query_conditions(&$conditions) { // foreach condition in condition group foreach ($conditions as &$condition) { // if condition is itself a condition group if (isset($condition['field']) && is_a($condition['field'], 'Drupal\Core\Database\Query\Condition')) { // call the helper function on it _recursively_alter_query_conditions($condition['field']->conditions()); } else { // check if we want to alter the condition and if so alter it _alter_query_condition($condition); } } } // separate helper function to determine if the condition is one we want to alter function _alter_query_condition(&$condition) { if (isset($condition['field']) && ($condition['field'] === 'node_search_index.word')) { $condition['value'] = "%{$condition['value']}%"; $condition['operator'] = 'LIKE'; } if (isset($condition['field']) && ($condition['field'] === 'node_search_dataset.data')) { // here we're using trim to eliminate both the unwanted whitespace and the '%' symbols, // which then get added back via string concatenation $condition['value'] = "%" . trim($condition['value'], " \t\n\r\0\x0B%") . "%"; } }
2.15.6. hook_views_pre_build
2.15.7. hook_views_pre_render
2.15.7.1. Change Global Custom Text
Change the field value with tokens in Views UI
function lili_views_pre_render(&$view) {
if ($view->name == 'lili_view' && $view->current_display == 'page') {
// Change value for specific rows
foreach ($view->result as $k => $r) {
if ($f = &$r->field_field_flag) {
$o = $f[0]['rendered']['#markup'];
$o = theme('lili_theme_flair', array('text'=>$o));
$f[0]['rendered']['#markup'] = $o;
}
// Taxonomy field
if ($f = &$r->field_field_category) {
$exclude_tids = array(1,2,3);
foreach ($f as $k => $v) {
if (!in_array($v['raw']['tid'], $exclude_tids)) {
// Remove from display
unset($f[$k]);
}
}
}
}
// Only change global custom text for a subset of contextual filter value
// The first context filter value
if ($view->args[0] == 'filter_value') {
// Check View UI Theme Information to get the Global Custom Text field ID
if (isset($view->field['field_thumbnail']->options['exclude'])) {
// Exclude display
$view->field['field_thumbnail']->options['exclude'] = 1;
}
if (isset($view->field['nothing']->options['alter']['text'])) {
// Change a field's CSS Class setting for all records
$view->field['nothing']->options['element_class'] = '';
$o = $view->field['nothing']->options['alter']['text'];
$view->field['nothing']->options['alter']['text'] = 'hello'.$o;
}
}
}
}
2.15.8. Custom View Filter hook_views_data_alter
http://precessionmedia.com/blog/how-create-custom-filter-handler-views
/** * Implements hook_views_data_alter(). */ function my_module_views_data_alter(&$data) { $data['node']['title_count']['title'] = 'Title word count'; $data['node']['title_count']['help'] = 'Count the number of words in titles.'; $data['node']['title_count']['filter']['handler'] = 'my_module_handler_filter_field_count'; }
my_module.info
files[] = my_module_handler_filter_field_count.inc
my_module_handler_filter_field_count.inc
class my_module_handler_filter_field_count extends views_handler_filter_numeric { function operators() { $operators = parent::operators(); // We won't be using regex in our example unset($operators['regular_expression']); return $operators; } // Helper function to return a sql expression // for counting words in a field. function field_count() { // Set the real field to the title of the node $this->real_field = 'title'; $field = "$this->table_alias.$this->real_field"; return "LENGTH($field)-LENGTH(REPLACE($field,' ',''))+1"; } // Override the op_between function // adding our field count function as parameter function op_between($field) { $field_count = $this->field_count(); $min = $this->value['min']; $max = $this->value['max']; if ($this->operator == 'between') { $this->query->add_where_expression($this->options['group'], "$field_count BETWEEN $min AND $max"); } else { $this->query->add_where_expression($this->options['group'], "($field_count <= $min) OR ($field_count >= $max)"); } } // Override the op_simple function // adding our field count function as parameter function op_simple($field) { $field_count = $this->field_count(); $value = $this->value['value']; $this->query->add_where_expression($this->options['group'], "$field_count $this->operator $value"); } }
In Views UI, add a filter with name "Title word count" with descript "Count the number of words in titles."
2.15.9. Better Exposed Filters Module d7:better exposed filters
If you want checkboxes or radio buttons instead of the Views Basic Exposed Form which is dropdown select.
Change BEF form hook_form_views_exposed_form_alter
function lili_form_views_exposed_form_alter(&$form, &$form_state) {
if (isset($form['#id'])) {
switch ($form['#id']) {
case 'views-exposed-form-your-form-name':
// ...
break;
}
}
}
The form id is views-exposed-form-view–machine-name-display-machine-name
2.15.10. Custom Views Plugin hook_views_plugins
2.15.11. Views API Order
hook_views_pre_view hook_views_pre_build hook_views_post_build hook_views_pre_execute hook_views_post_execute hook_views_pre_render hook_views_post_render
2.15.12. Embed View and Display
views_embed_view('view_name','display_id'); Doesn't display view title
$arg1 = $arg2 = '123';
print views_embed_view('view_name','display_id',$arg1,$arg2);
$my_view_name = 'magazines';
$my_display_name = 'block_1';
$my_view = views_get_view($my_view_name);
if (is_object($my_view)) {
$my_view->set_display($my_display_name);
$my_view->pre_execute();
echo $my_view->render($my_display_name);
}
2.15.13. View Theming d7:View_Theming
View, named foobar. Style: unformatted. Row styel: Fields. Display: Page.
- views-view-foobar–page.tpl.php
- views-view-page.tpl.php
- views-view–foobar.tpl.php
- views-view.tpl.php
- views-view-unformatted–foobar–page.tpl.php
- views-view-unformatted–page.tpl.php
- views-view-unformatted–foobar.tpl.php
- views-view-unformatted.tpl.php
- views-view-fields–foobar–page.tpl.php
- views-view-fields–page.tpl.php
- views-view-fields–foorbar.tpl.php
- views-view-fields.tpl.php
2.16. Ellipsis d7:ellipsis
Use truncate_utf8 for plain text string
truncate_utf8($string, $max_length, $wordsafe = FALSE, $add_ellipsis = FALSE, $min_wordsafe_length = 1)
This can be used in custom module. Trim first, then remove scraps of html
e.g. <a>fda</, <a>fda</a and <a>fda< will become <a>fda
'html' => true will close any unclosed html tags.
views_trim_text(array(
'max_length' => 200,
'word_boundary' => TRUE,
'ellipsis' => TRUE,
'html' => TRUE, // optional
), 'Long String');
// Output
Long String<span class="ellipsis">...</span>
Source code
function views_trim_text($alter, $value) { if (drupal_strlen($value) > $alter['max_length']) { $value = drupal_substr($value, 0, $alter['max_length']); // TODO: replace this with cleanstring of ctools if (!empty($alter['word_boundary'])) { $regex = "(.*)\b.+"; if (function_exists('mb_ereg')) { mb_regex_encoding('UTF-8'); $found = mb_ereg($regex, $value, $matches); } else { $found = preg_match("/$regex/us", $value, $matches); } if ($found) { $value = $matches[1]; } } // Remove scraps of HTML entities from the end of a strings $value = rtrim(preg_replace('/(?:<(?!.+>)|&(?!.+;)).*$/us', '', $value)); if (!empty($alter['ellipsis'])) { $value .= t('...'); } } if (!empty($alter['html'])) { $value = _filter_htmlcorrector($value); } return $value; } function _filter_htmlcorrector($text) { return filter_dom_serialize(filter_dom_load($text)); } function filter_dom_load($text) { $dom_document = new DOMDocument(); // Ignore warnings during HTML soup loading. @$dom_document->loadHTML('<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /></head><body>' . $text . '</body></html>'); return $dom_document; } function filter_dom_serialize($dom_document) { $body_node = $dom_document->getElementsByTagName('body')->item(0); $body_content = ''; if ($body_node !== NULL) { foreach ($body_node->getElementsByTagName('script') as $node) { filter_dom_serialize_escape_cdata_element($dom_document, $node); } foreach ($body_node->getElementsByTagName('style') as $node) { filter_dom_serialize_escape_cdata_element($dom_document, $node, '/*', '*/'); } foreach ($body_node->childNodes as $child_node) { $body_content .= $dom_document->saveXML($child_node); } return preg_replace('|<([^> ]*)/>|i', '<$1 />', $body_content); } else { return $body_content; } } // Adds comments around the <!CDATA section in a dom element. function filter_dom_serialize_escape_cdata_element($dom_document, $dom_element, $comment_start = '//', $comment_end = '') { foreach ($dom_element->childNodes as $node) { if (get_class($node) == 'DOMCdataSection') { // See drupal_get_js(). This code is more or less duplicated there. $embed_prefix = "\n<!--{$comment_start}--><![CDATA[{$comment_start} ><!--{$comment_end}\n"; $embed_suffix = "\n{$comment_start}--><!]]>{$comment_end}\n"; // Prevent invalid cdata escaping as this would throw a DOM error. // This is the same behavior as found in libxml2. // Related W3C standard: http://www.w3.org/TR/REC-xml/#dt-cdsection // Fix explanation: http://en.wikipedia.org/wiki/CDATA#Nesting $data = str_replace(']]>', ']]]]><![CDATA[>', $node->data); $fragment = $dom_document->createDocumentFragment(); $fragment->appendXML($embed_prefix . $data . $embed_suffix); $dom_element->appendChild($fragment); $dom_element->removeChild($node); } } }
2.17. Form
2.17.1. Form API
2.17.2. Form alter hooks
Called in order, from general to specific
hook_form_alter- for all forms
hook_form_BASE_FORM_ID_alter- Refer BASE_FORM_ID to
$form_state['build_info']['base_form_id'] hook_form_FORM_ID_alter- Refer FORM_ID to
$form['#id']
Set default value and hide a form from display or filling
// Entity reference form field $form['field_dealer'][LANGUAGE_NONE][0]['target_id']['#default_value'] = 123; $form['field_dealer'][LANGUAGE_NONE][0]['#printed'] = TRUE;
hook_form_FORM_ID_alter(&$form, &$form_state, $form_id)
2.17.3. Form IDs of system forms
Form
node- node new/edit
$form[''] $form['actions']['submit']['#submit'] = array('node_form_submit')
user_profile- page/user/<uid>
Alter
lili_node_form_alterhook_node_form_alter- node type (content type)
ad_listingand alter its node new/edit form lili_form_ad_listing_node_form_alterusinghook_form_FORM_ID_alter hookwhere FORM_ID isad_listing_node_form
Some modules alter the form before your custom alters. Such as field dependency module d7:module:conditional fields
2.17.4. Confirm Form
e.g. Confirm before delete
// Menu
$items['dealer_user/%/delete'] = array(
'title' => 'Delete a user',
'page callback' => 'drupal_get_form',
'page arguments' => array('lili_delete_dealer_user_confirm',1),
'access callback' => 'lili_user_has_delete_right',
'access arguments' => array(1, array('Dealer Admin')),
'type' => MENU_LOCAL_TASK,
);
function lili_delete_dealer_user_confirm($form, &$form_state, $id) {
$form['delete'] = array(
'#type' => 'value',
'#value' => $id,
);
return confirm_form($form,
t('Are you sure you want to delete this user?'), // Question
'dealer_manage_users', // Path to go to if No
t('This action cannot be undone.'), // Description
t('Delete Button'), // Yes: text
t('Cancel Button') // No: text
//, $name default 'confirm': internal name used to refer to the confirmation item.
);
}
function lili_delete_dealer_user_confirm_submit($form, &$form_state) {
if ($uid = $form_state['values']['delete']) {
$u = user_load($uid);
user_cancel();
drupal_set_message(t('The user account has been deleted!'));
}
$form_state['redirect'] = 'dealer_manager_users';
}
function lili_user_has_delete_right($uid = 0, array $roles) {
global $user;
// If user is anonymous
if (user_is_anonymous()) {
return FALSE;
}
if (in_array('administrator', $user->roles)) {
return TRUE;
}
if (array_intersect($roles, $user->roles)) {
return TRUE;
}
}
2.17.5. form_load_include
Include files whenever a form is loaded. form_load_include(&$form_state, $type, $module, $name = NULL) {} / if name is omitted, then $module.$type is loaded. / return The filepath of the loaded include file, or FALSE if the include file was / not found or has been loaded already. / file is always included even the form is cached. form_load_include($form_state, 'inc', 'tnt', 'tnt.pages');
2.17.6. Form Ajax
https://www.drupal.org/docs/7/api/javascript-api/ajax-forms-in-drupal-7
https://api.drupal.org/api/drupal/developer%21topics%21forms_api_reference.html/7.x#ajax
/**
* Ajax-enabled select element causes replacement of a set of checkboxes
* based on the selection.
*/
function ajax_example_autocheckboxes($form, &$form_state) {
$default = !empty($form_state['values']['howmany_select']) ? $form_state['values']['howmany_select'] : 1;
$form['howmany_select'] = array(
'#title' => t('How many checkboxes do you want?'),
'#type' => 'select',
'#options' => array(1 => 1, 2 => 2, 3 => 3, 4 => 4),
'#default_value' => $default,
'#ajax' => array(
'callback' => 'ajax_example_autocheckboxes_callback',
'wrapper' => 'checkboxes-div',
'method' => 'replace',
'effect' => 'fade',
),
);
$form['checkboxes_fieldset'] = array(
'#title' => t("Generated Checkboxes"),
// The prefix/suffix provide the div that we're replacing, named by
// #ajax['wrapper'] above.
'#prefix' => '<div id="checkboxes-div">',
'#suffix' => '</div>',
'#type' => 'fieldset',
'#description' => t('This is where we get automatically generated checkboxes'),
);
$num_checkboxes = !empty($form_state['values']['howmany_select']) ? $form_state['values']['howmany_select'] : 1;
for ($i = 1; $i <= $num_checkboxes; $i++) {
$form['checkboxes_fieldset']["checkbox$i"] = array(
'#type' => 'checkbox',
'#title' => "Checkbox $i",
);
}
$form['submit'] = array(
'#type' => 'submit',
'#value' => t('Submit'),
);
return $form;
}
/**
* Callback element needs only select the portion of the form to be updated.
* Since #ajax['callback'] return can be HTML or a renderable array (or an
* array of commands), we can just return a piece of the form.
*/
function ajax_example_autocheckboxes_callback($form, $form_state) {
return $form['checkboxes_fieldset'];
}
- #ajax['event']
- which jquery event triggers AJAX. Don't need to set. Default is fine.
- #ajax['callback']
- return HTML or renderable array to replace #ajax['wrapper']. By the time it gets called, the form is recalculated/rebuilt in PHP and you can return $form['the_field_name'] directly. The callback function can't change any state or form settings. Do it in form builder functions: hook_form, hook_form_alter. Callback function is called after all the form processing fuctions.
- #ajax['wrapper']
- HTML container.
2.17.7. Form field loaded with terms of a taxonomy fields d7:form:taxonomy field
$parts_makes = ['0' => t('Make')]; // First option
$_terms = taxonomy_allowed_values(field_info_field('field_parts_make'));
// [123] => term123_name, [345] => term345_name, ...
if ($language->language == 'en') {
$parts_makes += $_terms;
}
else {
$parts_make_terms = $_terms;
foreach ($parts_make_terms as $tid => $name) {
$i18n_object = i18n_get_object('taxonomy_term', $tid);
if ($translated_term = $i18n_object->localize($language->language)) {
$parts_makes[$tid] = (!empty($translated_term->name)) ? $translated_term->name : $parts_make_terms[$tid];
}
}
}
$form['make'] = [
'#type' => 'select',
'#options' => $parts_makes,
];
2.18. Action
hook_info_action return ['action_name' => $an_action];
$an_action :: array
- 'permissions'
- this is specific for View Bulk Operations (VBO). e.g. array('switch users')
- 'behavior'
- array. VBO uses one of views_property, changes_property, creates_property, deletes_property
2.19. Field API d7:api:field
When a field associates with an entity, a widget can be specified A field with an entity can have one formatter for each view mode.
2.19.1. hook_field_is_empty d7:api:field:hook_field_is_empty
modules/field/field.api.php hook_field_is_empty($item, $field) /modules/field/modules[fieldtype]/[fieldtype].module Or any modules that implements hook_field_is_empty
Core fieldtype:
- field_sql_storage
- list
- number
- options
- text
- taxonomy
- image
- file
$info = field_info_field($field_name); $function = $info['module'] . '_field_is_empty'; if (function_exists($function)) { $value = field_get_items('node', $node, $field_name); $is_empty = $function($value, $info); }
2.19.2. Field Formatter d7:api:field:formatter
Content Type > Manage Display > Format for a field hook_field_formatter_info tells Drupal a new field formatter that will apply to what field types. If a formatter has a setting, a summary and a form needs to be defined in order to show a gear icon next to summary in Manage Display
hook_field_formatter_info :: hook_field_formatter_info_alter
function text_field_formatter_info() {
return array(
'text_default' => array(
'label' => t('Default'),
'field types' => array('text', 'text_long', 'text_with_summary'),
),
'text_plain' => array(
'label' => t('Plain text'),
'field types' => array('text', 'text_long', 'text_with_summary'),
),
'text_trimmed' => array(
'label' => t('Trimmed'),
'field types' => array('text', 'text_long', 'text_with_summary'),
'settings' => array('trim_length' => 600),
),
'text_summary_or_trimmed' => array(
'label' => t('Summary or trimmed'),
'field types' => array('text_with_summary'),
'settings' => array('trim_length' => 600),
),
);
}
hook_field_formatter_settings_summary
function text_field_formatter_settings_summary($field, $instance, $view_mode) {
$display = $instance['display'][$view_mode];
$settings = $display['settings'];
$summary = '';
if (strpos($display['type'], '_trimmed') !== FALSE) {
$summary = t('Trimmed limit: @trim_length characters', array('@trim_length' => $settings['trim_length']));
}
return $summary;
}
hook_field_formatter_settings_form
function text_field_formatter_settings_form($field, $instance, $view_mode, $form, &$form_state) {
$display = $instance['display'][$view_mode];
$settings = $display['settings'];
$element = array();
if (strpos($display['type'], '_trimmed') !== FALSE) {
$element['trim_length'] = array(
'#title' => t('Trimmed limit'),
'#type' => 'textfield',
'#field_suffix' => t('characters'),
'#size' => 10,
'#default_value' => $settings['trim_length'],
'#element_validate' => array('element_validate_integer_positive'),
'#description' => t('If the summary is not set, the trimmed %label field will be shorter than this character limit.', array('%label' => $instance['label'])),
'#required' => TRUE,
);
}
return $element;
}
hook_field_formatter_view :: use settings to return result (HTML). Use hook_field_formatter_prepare_view to add info for field values being displayed.
function text_field_formatter_view($entity_type, $entity, $field, $instance, $langcode, $items, $display) {
$element = array();
switch ($display['type']) {
case 'text_default':
case 'text_trimmed':
foreach ($items as $delta => $item) {
$output = _text_sanitize($instance, $langcode, $item, 'value');
if ($display['type'] == 'text_trimmed') {
$output = text_summary($output, $instance['settings']['text_processing'] ? $item['format'] : NULL, $display['settings']['trim_length']);
}
$element[$delta] = array('#markup' => $output);
}
break;
case 'text_summary_or_trimmed':
foreach ($items as $delta => $item) {
if (!empty($item['summary'])) {
$output = _text_sanitize($instance, $langcode, $item, 'summary');
}
else {
$output = _text_sanitize($instance, $langcode, $item, 'value');
$output = text_summary($output, $instance['settings']['text_processing'] ? $item['format'] : NULL, $display['settings']['trim_length']);
}
$element[$delta] = array('#markup' => $output);
}
break;
case 'text_plain':
foreach ($items as $delta => $item) {
$element[$delta] = array('#markup' => strip_tags($item['value']));
}
break;
}
return $element;
}
Usage
$node = node_load($view->result[0]->_field_data['nid']['entity']->nid);
$settings = array(
'label' => 'hidden',
'type' => 'text_summary_or_trimmed',
'settings' => array('trim_length' => 800),
);
$node_summary = field_view_field('node', $node, 'body', $settings);
print render($node_summary);
2.19.3. field_get_items d7:field_get_items
Return the field items in the language they currently would be displayed. $items = field_get_items($entity_type, $entity, $field_name, $langcode = NULL)
If there is no field associated with $entity or the field is not a field belongs to $entity, it will return false.
// field_get_items($entity_type, $entity, $field_name, $langcode = NULL) // field_get_items('node', $node, 'field_my_field_name' ) if (!empty(field_get_items('node', $node, 'field_my_field_name'))) { // field has at least one value } // or use this to determine how many values a field has function hasValues($field_name,$entity, $entity_type='node', $langcode = NULL) { $r = [ 'has' => FALSE, 'count' => 0 ]; $value = field_get_items($entity_type, $entity, $field_name, $langcode); if (is_array($value)) { $r = [ 'has' => true, 'count' => count($value) ]; } return $r; }
To check if a field value is empty d7:api:field:hook_field_is_empty
After checking it has values, then you can render($content['field_name']) in node.tpl.php
2.19.3.1. Image field
fid => "768" uid => "1" filename => "splash-photo.jpg" uri => "public://splash-photo.jpg" filemime => "image/jpeg" filesize => "129265" status => "1" timestamp => "1518792000" alt => "" title=> "" width => "727" height => "426"
2.20. Database d7:database
2.20.1. Close Comment for Existing Nodes
0 = No, 1 = Closed (read only), 2 = Open (Read/Write)
UPDATE node, node_revision SET node.comment = 1, node_revision.comment = 1 WHERE node_revision.nid = node AND node.type = 'article'
2.20.2. Structure
2.20.2.1. node
- nid PK
- int(10) unsigned
- vid
- The current node_revision.vid version id
- uid
- int(11) d:0 user/author id
- type
- varchar(32) d:'' e.g. article
- title
- varchar(255) d:''
- created
- int(11) d:0 unix timestamp
- changed
- int(11) d:0 unix timestamp
- comment
- int(11) d:0
2 = open (read/write), 1 = closed (read only), 0 = no - sticky
- int(11) d:0 Boolean to display node at the top of query result
- promote
- int(11) d:0 Boolean to show on homepage
- tnid
- int(10) unsigned d:0 translation set id
- translate
- int(11) d:0 Boolean to mark translation is needed
2.20.2.2. field_config
- id PK
- int 11
- field_name
- vchar 32 e.g. field_mileage_units
- type
- vchar 128
- text
- datetime
- link_field
- phone
- list_text
- list_boolean
- file
- taxonomy_term_reference
- entityreference
- module
- vchar 128
- text
- date
- link
- phone
- list
- file
- taxonomy
- entityreference
- active
- tinyint 4. bool NOT NULL
- storage_type
- vchar 128 e.g. field_sql_storage
- storage_module
- vchar 128 e.g. field_sql_storage
- locked
- tinyint 4 don't know..
- data
- longblob NOT NULL serialized data containing the field properties
- cardinality
- tinyint 4 default 0 don't know..
- translatable
- tinyint 4 default 0 don't know..
- deleted
- tinyint 4 default 0 don't know..
2.20.2.3. field_data_body
- Common fields
- entity_type PK
- varchar(128) d:'' this field is attached to e.g. node
- entity_id PK
- int(10) unsigned e.g. node id
- bundle
- varchar(128) d:'' e.g. node type article
- deleted PK
- tinyint(4) d:0 not deleted
- language PK
- varchar(32) d:'' e.g.
und - delta PK
- int(10) unsigned used for ulti-value fields, sequence number, start as 0
- revision_id PK
- int(10) unsigned
- Unique fields
- body_value
- longtext
- body_summary
- longtext
2.20.2.4. field_data_[field_my_text_or_longtext]
- Text
field_data_[field_my_text]_value- varchar(255)
field_data_[field_my_text]_format- varchar(255)
- Longtext
- longtext
- varchar(255)
2.20.2.5. field_data_[field_my_date_unix_timestamp]
field_data_[field_my_date_unix_timestamp]_value- int(11)
2.20.2.6. field_data_[field_custom_term_reference]
- field_custom_term_referecen_tid
- int
2.20.2.7. field_data_field_mileage_units list_text
- entity_type PK I-A
- vchar 128 default '' NOT NULL The entity type this data is attached to
- node
- (no term)
- bundle I-A:: vchar 128 default '' NOT NULL The field instance bundle to which this row belongs, used when deleting a field instance
- my_node_type
- deleted PK I-A
- tinyint 4 default 0 NOT NULL
- entity_id PK I-A
- int 10 NOT NULL the entity id this data is attached to.
- revision_id I-A
- int 10 or NULL if the entity type is not versioned
- language PK I-A
- vchar 32 default '' NOT NULL The language for this data item
- delta PK
- int 10 NOT NULL the sequence number for this data item, used for multi-value fields
- field_mileage_units_value I-A
- vchar 255 NULL
2.20.2.8. field_data_field_contributor entity reference
- field_contributor_target_id
- int
2.20.2.9. taxonomy_term_data
- tid*
- int
- vid
- int, vocabulary id. see taxonomy_vocabulary
- name
- string, term name
- weight
- int
2.20.2.10. taxonomy_vocabular
vid* :: name :: machine_name
2.20.2.11. url_alias
pid* :: int, path id source :: string, e.g. node/2, taxonomy/term/11 alias :: string language :: string, e.g. und
2.20.2.12. Export Query
SELECT n.title,
b.body_value,
GROUP_CONCAT(t.name SEPARATOR ', ') AS Category,
IFNULL(na.title,'') AS author,
CONCAT('http://domain.com/',u.alias) as url,
FROM_UNIXTIME(n.created) as post_date
FROM node AS n
INNER JOIN field_data_body b ON b.entity_id=n.nid AND b.delta=0
INNER JOIN field_data_field_category c ON c.entity_id=n.nid AND c.deleted=0
LEFT JOIN taxonomy_term_data t ON t.tid = c.field_category_tid
INNER JOIN field_data_field_contributor a ON a.entity_id=n.nid AND a.deleted=0
LEFT JOIN node as na ON na.nid=a.field_contributor_target_id
INNER JOIN url_alias u ON u.source = concat('node/',n.nid)
WHERE FROM_UNIXTIME(n.created)
BETWEEN '2014-01-01 00:00:00' AND '2014-12-31 23:59:59'
AND n.type ='article'
GROUP BY n.nid
ORDER BY n.created DESC
Export taxonomy vocabular
SELECT
t.tid AS tid,
t.name,
h.parent AS parent_tid,
t.weight,
CONCAT('http://yourweb.com/', u.alias) AS url
FROM taxonomy_vocabulary v
LEFT JOIN taxonomy_term_data t ON t.vid = v.vid
LEFT JOIN taxonomy_term_hierarchy h ON h.tid = t.tid
LEFT JOIN field_data_field_mega_title mega_title
ON mega_title.entity_id = t.tid
LEFT JOIN field_data_field_mega_story_title mega_story_title
ON mega_story_title.entity_id = t.tid
LEFT JOIN field_data_field_landing_page_type splash
ON splash.entity_id = t.tid
LEFT JOIN url_alias u ON u.source = concat('taxonomy/term/', t.tid)
WHERE
0 = 0
AND v.machine_name = 'articles'
ORDER BY h.parent, t.weight
2.20.2.13. Export File URI, file_managed d7:db:file uri
SELECT n.nid, n.title ,ds_i_fm.uri AS splash_image -- e.g. public://abc.jpg ,REGEXP_REPLACE(ds_i_fm.uri, "^(s3://)(.*)", 'http://s3.amazonaws.com/abc.com/\\2') AS splash_image, -- s3://abc.jpg FROM node AS n LEFT JOIN field_data_field_image ds_i ON ds_i.entity_id=n.nid AND ds_i.deleted=0 LEFT JOIN file_managed ds_i_fm ON ds_i_fm.fid = ds_i.field_image_fid WHERE n.type = 'article' AND n.nid = 123
2.20.2.14. file_usage
fid id :: e.g. node id
2.20.2.15. system
2.20.2.16. users
*uid :: int(10) name :: pass :: mail :: email init :: initial email created :: int(11), unix time access :: int(11), unix time. previous time user accessed the site login :: int(11), unix time status :: tinyint(4)
2.20.2.17. users_role
uid rid
2.20.2.18. role
rid* name weight
2.21. Query
Use db_query as much as possible. db_select is slow because it calls alter hooks. Both can only be looped once using foreach
Use db_query or db_select to get all nids, load all nodes at once and then manipulate fields.
db_query is meant for simple select query. db_select is dynamic query
2.21.1. db_select
$q = db_select( 'node', 'n' ); $q->join( 'field_data_body', 'b', 'n.nid=b.entity_id AND b.delta=0' ); $q->innerJoin( 'field_data_field_category', 'nc' , 'n.nid=nc.entity_id AND nc.deleted=0' ); $q->innerJoin( 'taxonomy_term_hierarchy', 'tH', 'tH.tid=nc.field_category_tid' ); if ( ! is_null( $termID ) ) { $articleCats = array( $termID ); } $q->fields( 'n', array( 'nid', 'title', 'created' ) ) ->fields( 'b', array( 'body_summary', 'body_value' ) ) ->condition( 'n.status', 1 )// Published. ->condition( 'n.type', 'article' )// article not page. ->condition( db_or() ->condition( 'nc.field_category_tid', $articleCats, 'IN' ) ->condition( 'tH.parent', $articleCats, 'IN' ) ) ->range( $i_start, $i_length ) ->groupBy( 'n.nid' ) ->orderBy( 'created', 'DESC' ); // debug a query print_r($query->__toString()); print_r($query->arguments()); $r = $q->execute(); $nids = [ ]; foreach ( $result as $n ) { $nids[] = $n->nid; } $nodes = node_load_multiple( $nids ); foreach ( $nodes as $node ) { $nodeID = $node->nid; $nodeTitle = $node->title; $nodeBody = lili_getFieldSafe( $node, 'body' ); $nodeURL = url( "node/" . $node->nid ); $nodeSplashObj = field_get_items( 'node', $node, 'field_splash_image' ); if ( is_array( $nodeSplashObj ) ) { // $nodeSplashObj['#item']['uri']; // $nodeSplashObj['#item']['width']; // $nodeSplashObj['#item']['height']; } // Entity Reference $nodeFlag = field_get_items('node',$nodeObj,'field_flag'); if ($nodeFlag) { $nodeFlag = field_view_value('node', $node, 'field_flag', $nodeFlag[0]); $nodeFlag = $nodeFlag['#markup']; $nodeFlag = theme_infscroll_flair(array('text'=>$nodeFlag)); } else { $nodeFlag = ''; } } function lili_getFieldSafe( $node, $field ) { $s = field_get_items( 'node', $node, $field ); if ( $s ) { $s = ( $s[0]['safe_value'] ) ? $s[0]['safe_value'] : ''; } else { $s = ''; } return $s; }
2.21.2. db_query
https://www.drupal.org/docs/7/api/database-api/static-queries https://api.drupal.org/api/drupal/includes%21database%21database.inc/function/db_query/7.x
$query = db_query("SELECT nid, title FROM {node}"); $records = $query->fetchAll(); // get all records into an indexed array of stdClass $records = $query->rowCount(); foreach ($records as $record) { // Do something. } // placeholders should not be escaped or quoted $result = db_query("SELECT nid, title FROM {node} WHERE created > :created", array( ':created' => REQUEST_TIME - 3600, )); db_query("SELECT * FROM {node} WHERE nid IN (:nids)", array(':nids' => array(13, 42, 144))); // query options, use 'target' and 'fetch' only 'target' => 'default', // or 'slave' 'fetch' => PDO::FETCH_OBJ // or PDO::FETCH_ASSOC, PDO::FETCH_NUM, PDO::FETCH_BOTH or a string of a class name $sql = "SELECT name, quantity FROM goods WHERE vid = :vid"; $result = db_query($sql, array(':vid' => $vid)); if ($result) { while ($row = $result->fetchAssoc()) { // fetch next row as an associative array // vs fetch next row as an object $result->fetchObject() // Do something with: // $row['name'] // $row['quantity'] } } // Get the first column result set as one single array $nids = $result->fetchCol(); // Column number can be specified otherwise default to first column $nids = $result->fetchCol(2); // say first column is node id, then load all nodes $nodes = node_load_multiple( $nids );
Other fetch usage
// Fetch data from specific column from next row // Defaults to first column if not specified as argument $data = $result->fetchColumn(1); // Grabs the title from the next row // Retrieve all records as stdObjects into an associative array // keyed by the field in the result specified. // (in this example, the title of the node) $result->fetchAllAssoc('title'); // Retrieve a 2-column result set as an associative array of field 1 => field 2. $result->fetchAllKeyed(); // Also good to note that you can specify which two fields to use // by specifying the column numbers for each field $result->fetchAllKeyed(0,2); // would be nid => created $result->fetchAllKeyed(1,0); // would be title => nid
2.21.3. db_delete
Same syntax as db_select
$num_deleted = db_delete('node')
->condition('nid', 5)
->execute();
Clear cache is needed if field_data_field_xxx are deleted. Delete a required field for node (nid) d7:db:delete required field
$q = db_delete('field_data_field_trailer_make');
$q->condition('entity_id', $node->nid);
$r = $q->execute();
2.21.4. hook_query_TAG_alter hook_query_TAG_alter
function mymod_query_node_is_not_tagged_alter(QueryAlterableInterface $query) {
$query->leftJoin('field_data_field_tags', 'o', 'node.nid = o.entity_id AND o.entity_type = :entity_type');
$query->isNull('o.field_tags_tid');
}
2.22. Entity
Node, user, taxonomy are all entities.
2.22.1. EntityFieldQuery d7:efq
Use this to get entities of one type that have certain:
- entity properties (propertyCondition: status, changed, etc)
- field values (fieldCondition)
- entity meta data (entityCondition: bundle, entity_type, entity_id and revision_id)
- It's essentially db_select except EFQ can't do joins
- It's a lot slower
- But it provides simple syntax
- It's good for quick filtering entities by basic field values and return entity ids
- Preferred over db_select
- EFQ can return any type of entities including nodes of various types
- EFQ can only return essential fields such as node nid, vid and type
- Use EFQ result nids and node_load_multiple or entity_load
- Use EFQ result nids and get values of a field for all nodes (load only one field for all nodes)
- If you want custom fields and node->title, better to use node_load_multiple or entity_load
- By default without addMetaData, EFQ requires permissions of all the fields!
$_q = new EntityFieldQuery();
$_r = $_q
->entityCondition('entity_type', 'node') // taxonomy_term, user
->entityCondition('bundle', 'newsletter') // node type or content type
->fieldCondition('field_active','value',1,'=') // boolean field, only 1 or 0
->fieldCondition('field_news_publishdate', 'value', $year . '%', 'like') // Text LIKE
->fieldOrderBy('field_ns_newsletter_unique_id', 'value', 'DESC') // Sort. refer to propertyOrderBy for created date
->fieldCondition('field_a_term_reference_field', 'tid', 123) // field Term Reference
->fieldCondition('field_a_term_reference_field', 'tid', $tids) // field Term Reference multiple has at least one value in array $tids
->fieldCondition('field_dealer', 'target_id', 12345, '=') // field Entity Reference, single value
->fieldCondition('field_contacts', 'target_id', 12345) // field Entity Reference, multiple values have one 12345
/* NOT IN
12345 not in mutliple target_id: you have to do one query to show all and do one with 'IN' and
get the difference
$_r_all = (!empty($_r_all['node'])) ? $_r_all['node'] : array();
$_r_in_12345 = (!empty($_r_in_12345['node'])) ? $_r_in12345['node'] : array();
$_r = array_diff_key($_r_all, $_r_in_12345);
*/
->propertyCondition('status',1)
// status, type, nid, uid, rui, type, created
// search for "Implements hook_entity_info()" will show all entity properties
// Operators
// Default '='
// <>, >, >=, <, <=
// STARTS_WITH, CONTAINS
// IN, NOT IN (does not work)
// BETWEEN :: propertyCondtion('created', array($start, $end), 'BETWEEN')
->propertyOrderBy('created', 'DESC') // sort by created date
->range(0,10)
// Random order. hook_query_TAG_alter
->addTag('random')
/* If you run EFQ in Drupal Cron, by default the user is anonymous
You should run EFQ as a user
*/
->addMetaData('account', user_load(1))
->execute();
/* array(
'node' => // follow the entity_type: e.g. node, taxonomy_term, user
array(
28912 => obj(nid => 28912) // could be uid, nid, tid
...
)
)
*/
// or $_r['user']
if (!empty($_r['node'])) {
$nodes = node_load_multiple(array_keys($_r['node']));
// If you just want to load a handful of fields, you don't have to load the whole nodes
// $stories = $_r['node'];
// Get all fields of the entity type first
// $fields = field_info_instances('node', 'story'); // node type is story
// Get the field id
// $field_id = $fields['field_story_image']['field_id'];
// Attach the field to the entities loaded by EFQ
// field_attach_load('node', $stories, FIELD_LOAD_CURRRENT,
// array('field_id' => $field_id));
// All the values of field_story_image of all nodes
// $output = field_get_items('node', $stories, 'field_story_image');
}
2.22.2. Update a field of an entity only d7:field_attach_update
Recommend d7:entity_metadata_wrapper node_save could invoke other hooks and hence cause errors in saving a field value
$node = node_load($nid); $node->field_fieldname[LANGUAGE_NONE][0]['value'] = 'some value'; node_save($node);
Use field_attach_update() instead
$node = node_load($nid);
$node->field_fieldname[LANGUAGE_NONE][0]['value'] = 'some value';
unset($node->field_unwanted_1); // remove unwanted fields
field_attach_update('node', $node);
Without node_load
$node = new stdClass();
$node->id = $id_value; // node id
$node->type = $content_type; // aka content type
$node->field_fieldname[LANGUAGE_NONE][0]['value'] = 'some value';
field_attach_update('node', $node);
May be better to call field_attach_presave which node_save does
field_attach_presave('node', $node); // call hook_field_attach_presave
field_attach_update('node', $node);
2.22.3. entity_metadata_wrapper d7:entity_metadata_wrapper
Get a field value of an entity (node)
- Add a wrapper first
$_wrapper=entity_metadata_wrapper('node',$node); - $node can be a node/entity object or the node/entity id.
- Entity API module is needed.
- List all fields using
$_wrapper->getPropertyInfo(); - Get a specific field
$_wrapper->field_name->value();or->raw();
// Unified way of getting $node->title, $user->name, ... $wrapper->label(); or $wrapper->title->value(); // Unified way of getting $node->nid, $user->uid, ... $wrapper->getIdentifier(); // Unified way of getting $node->type, ... $wrapper->getBundle(); // Text field $old_value = $wrapper->field_my_text->value(); if (isset($old_value)) { // non-null, could be empty or 0 } $wrapper->field_my_text = 'new value'; // Longtext $wrapper->body->value(); // Array of: value, safe_value, format, and optionally summary + safe_summary. $wrapper->body->value->value(); // Filtered value. $wrapper->body->value->raw(); // Unfiltered value. $wrapper->body->format->value(); // The selected text format. $wrapper->body->value = 'new value'; // Link $wrapper->field_my_link->value(); // Array of: url, title, attributes. $wrapper->field_my_link->url->value(); // Hyperlink destination. $wrapper->field_my_link->title->value(); // Hyperlink text. $wrapper->field_my_link->attributes->value(); // Array of: target, title, etc. $wrapper->field_my_link->url = 'http://mediacurrent.com'; $wrapper->field_my_link->title = 'Do Drupal Right'; // Radio and Checkbox $v = $wrapper->field_my_radio->value(); // on = 1, off = 0 $info = field_info_field('field_my_radio'); // on = New, off = Used $allowed_values = $info['settings']['allowed_values']; $label = $allowed_values[$v]; // on = New, off = Used // Delete a value, unset a field, taxonomy in selectlist is different $wrapper->field_foobar = NULL; // Save $wrapper->save(); // Select list of taxonomy or entity reference, General Usage Append index `$term = $_wrapper->field_taxonomy_a[0]->value();` If only one value is allowed, it doesn't have index. Loop foreach ($_wrapper->field_list_of_values as $item) { // do something } // Taxonomy in select list For taxonomy term field which can be multiple values, you still need to specify the index `$term = $_wrapper->field_taxonomy_a[0]->value();` // this might be wrong..
Multiple Tax Terms
$a = $wrapper->$field->value(); if (isset($a)) { echo $a[0]->name; echo $a[0]->tid; }
Single Tax Term
$a = $wrapper->$field->value(); if (isset($a)) { echo $a->name; echo $a->tid; }
Multiple Files
$a = $wrapper->$field->value(); if (isset($a)) { $r=$a; foreach ($r as $k => $v) { $r[$k]['url'] = file_create_url($r[$k]['uri']); // convert URI to public URL // Drupal doesn't provide public 'url' } echo $r[0]['uri']; echo $r[1]['uri']; }
Term id of the first value of a multiple value field $tid = $_wrapper->field_taxonomy_a[0]->raw();
For taxonomy term field which can only have one value:
Term ID: $_wrapper->field_tax_1->raw();
Check empty if (isset($tid))
Term name: $_wrapper->field_tax_1->name->value();
Or in a multi value list $v = $_wrapper->field_tax_1[0]->value(); echo $v->name;
Add a term or entity reference: $_w->field_tax_1[] = 42; $_w->save();
Set as a whole $_w->field_tax_1->set(array(42,43)); $_w->save();
Change a term for a single tax field $_w->field_tax_1->set(42); $_w->save();
A required field can't be unset. Refer to d7:db:delete required field Unset a term for a single tax field unset($_w->field_tax_1) Unset a term for a multi tax field
foreach($wrapper->field_foobar as $delta => $item) { if ($item->nid->value() == $my_id) { unset($wrapper->field_foobar[$delta]); $wrapper->save(); break; } }
Set empty value for tax term field
$wrapper->field_example_multiple->set(); $wrapper->field_example_multiple = array(); $wrapper->field_example_multiple = NULL;
Loop
foreach ($_wrapper->field_industry as $taxonomy_wrapper) { $industry = $taxonomy_wrapper->tid->value(); }
A list of text
// All values in a list text as an array $wrapper->field_list_text->value(); foreach ($wrapper->field_taxonomy_terms->getIterator() as $delta => $term_wrapper) { // $term_wrapper may now be accessed as a taxonomy term wrapper. $label = $term_wrapper->name->value(); // list text, delta is 0, 1 // List text value $term_wrapper->value(); }
If the list of text field only allows one value, then the value is
$w->field_list_text->value(); The same as getting a text field
// Entity reference in select list
Single value. Tid is $tid = $wrapper->field_list_entity->raw();
Check empty if (isset($tid)) { echo $wrapper->field_list_entity->value()->title}
Multiple values.
https://www.drupal.org/documentation/entity-metadata-wrappers http://www.mediacurrent.com/blog/entity-metadata-wrapper
2.23. Entityform
D7 uses entityform module. D8 is called EForm. Load an entityform
// Load the module file. Optional
// module_load_include('inc', 'entityform', 'entityform-admin');
$_contact_form = entityform_empty_load('entityform_id');
$mode = 'submit';
// Default. Other: 'edit'
$form_context = 'page';
// Default. Ohter: 'embedded'
$_renderable_form = entityform_form_wrapper($_contact_form, $mode, $form_context );
Try to prefill entityform in hook_form_FORM_ID_alter(). Say the entityform is named dealer_contact The hook is lili_form_dealer_contact_entityform_edit_form_alter
Use modal in CTools, you will need t:
- Entityform
- Modal operations
- Modal forms (with ctools)
- Enable the Modal entityforms module
- Create a link (e.g. in View)
<a href="/modal/entityform/YOUR_FORM_IDENTIFIER/nojs/0" class="ctools-use-modal ctools-modal-modal-popup-small">Open Entity Form in Popup!</a>
2.24. User
2.24.1. Get user fields
global $user
$account = user_load($user->uid) to load all fields
or use d7:entity_metadata_wrapper to get a couple fields
$user_wrapper = entity_metadata_wrapper('user', $user); drupal_set_message('<pre>'.var_export($user_wrapper->field_node->raw(),1).'</pre>'); // Raw value drupal_set_message('<pre>'.var_export($user_wrapper->field_node->value(),1).'</pre>');
2.24.2. user_access d7:user_access
user_access($string, $account = NULL)
$account
- NULL. current logged in user
2.24.3. user_is_logged_in, user_has_role
user_is_logged_in && user_has_role(3) :: check if current user is admin
anonymous :: 1 authenticated user :: 2 administrator :: 3
2.25. Menu d7:menu
2.25.1. hook_menu: 'page callback'
- You can print or echo some rendered HTML and then
drupal_exit()orexit()- drupal_exit()
- prevents hook_exit() from running
- (no term)
- Or print and echo some HTML and
return NULL; - (no term)
- No header or footer will be added. The page is pure what you output here.
- You can return a renderable array. Header and footer will be included
- You may theme the form by
$form['#theme'][] = 'custom_theme_name';andreturn $form return theme('custom_theme_name', ['vars'=>...]);- In page callback function, you can manipulate some variables used in
html.tpl.phpandpage.tpl.php- Such as change the page title
drupal_set_title
- Such as change the page title
- Refer to d7:functions
2.25.2. hook_menu: 'file'
You can local the page callback function in a different file. e.g. 'lili.pages.inc' The path is the current module path
2.25.3. hook_menu: 'type'
- MENU_CALLBACK
- A hidden internal callback. Best for return plain text in API calls
- MENU_LOCAL_ACTION
- An action specific to the parent, usually rendered as a link
- MENU_LOCAL_TASK
- A task specific to the parent item, usually rendered as a tab
- MENU_NORMAL_ITEM
- Shown in menu and breadcrumbs
- MENU_SUGGESTED_ITEM
- A normal menu item, hidden until enabled by an administrator
2.25.4. hook_menu: 'access callback', 'access arguments'
- 'access callback'
- default is 'user_access' d7:user_access
- custom callback must return boolean
- Often used callbacks: 'user_is_logged_in'
- TRUE is for anyone to access
- 'access arguments'
- e.g. ['administer nodes']
2.25.5. Embed a View Page
function lili_menu() { $items['path-different-from-view/%'] = array( 'title' => t('page title'); 'page callback' => 'views_embed_view', 'page arguments' => array('view_id', 'dispaly_id', 1), 'type' => MENU_CALLBACK, 'access callback' => TRUE, ); return $items; }
2.25.6. Array to CSV
page callback
drupal_add_http_header('Content-Type', 'text/csv; utf-8'); drupal_add_http_header('Content-Disposition', 'attachment;filename=csvfile.csv'); $fp = fopen('php://output', 'w'); fputcsv($fp, [$year. ' Ranking of Top 100 Carriers']); $_columns = '#,Company,Web Site,Total,Trucks,Tractors,Trailers,O/OS,Employees'; $_columns = explode(',',$_columns); fputcsv($fp, $_columns); foreach ($data as $d) { $line = [ ]; $line[] = $d->this_rank; $line[] = $d->company_name; $line[] = $d->web_address; $line[] = $d->total_vehicles; $line[] = $d->trucks; $line[] = $d->tractors; $line[] = $d->trailers; $line[] = $d->owner_operators; $line[] = $d->full_time_employees; fputcsv( $fp, $line ); } fclose($fp); drupal_exit();
2.25.7. Json
function lili_menu() { $items['newsletter/archive/%/%'] = array( 'title' => "", 'page callback' => '_lili_archive_by_newsletter_json', 'page arguments' => array(2,3), 'access callback' => TRUE, 'type' => MENU_CALLBACK, ); return $items; } function _lili_archive_by_newsletter_json($pub_id,$newsletter_name, $page = 1, $items = 20) { drupal_add_http_header('Access-Control-Allow-Origin','*'); // CORS $_result['total'] = '...'; $_result['items'] = array('...'); drupal_json_output($_result); //return 'hello'; }
2.26. Taxonomy
A node or an entity has a taxonomy term means the node has a category. Categories may have hierarchy.
2.26.1. Get taxonomy terms of a node, ready for display with language
$categories = field_get_items('node', $node, 'field_category');
array(
0 => array(
'tid' => 17,
'taxonomy_term' => obj
),
...
)
2.26.2. Get multiple terms by id, taxonomy_term_load_multiple
$tids = array(1,2,3); // Default:array() $conditions = array(); // Default taxonomy_term_load_multiple($tids, $conditions); array( 17 => obj( tid => '17', ... ) ... )
2.26.3. Get term by name, taxonomy_get_term_by_name
// taxonomy_get_term_by_name($name, $vocabulary = NULL) $_limit_to_vocab = 'vocab_name'; // Default NULL. $term_is_parent = taxonomy_get_term_by_name('video', $_limit_to_vocab); // load the term's field values $term_is_parent_id = (!empty($term_is_parent)) ? key($term_is_parent) : 0; $field_my_field = ($term_is_parent_id) ? field_get_items( 'taxonomy_term', $term_is_parent[$term_is_parent_id], 'field_my_field'); // refer to d7:field_get_items if (!empty($field_my_field)) { foreach ($field_my_field as $k => $v) { echo $v['value']; } }
2.26.4. Get all children of a term, taxonomy_get_children
Get a term that is parent
$_limit_to_vocab = 'vocab_name'; // Default NULL.
$term_is_parent = taxonomy_get_term_by_name('video', $_limit_to_vocab);
// Get the first item in array
$term_is_parent = reset($term_is_parent);
echo $term_is_parent->tid; // parent term id
// Get all children of a term
$children = taxonomy_get_children($term->tid);
$children_tids = array_keys($children);
2.26.5. Get a vocabulary by name and all terms
$_vocab = taxonomy_vocabulary_machine_name_load('vocab_name');
// Get all terms of a vocabulary without extra fields
$_terms = taxonomy_get_tree($_vocab->vid);
// Get term ids only of all terms under a vocabulary
$_q = new EntityFieldQuery();
$_term_ids = $_q
->entityCondition('entity_type', 'taxonomy_term')
->propertyCondition('vid', $_vocab->vid)
->execute();
foreach ($_term_ids['taxonomy_term'] as $term) {
$term->id;
}
// Get all fields of all terms under a vocabulary
$_terms_with_all_fields = entity_load('taxonomy_term', FALSE, array('vid' => $_vocab->vid));
/* array(
17 => obj(
tid => '17',
...
),
...
)
*/
// Or
$parent = 0; // Default
$max_depth = NULL; // Default
$load_entities = TRUE; // Default:False
$_terms_with_all_fields = taxonomy_get_tree($_vocab->vid, $parent, $max_depth, $load_entities);
array(
0 => obj(
tid => '17',
...
)
...
)
2.26.6. Get terms of a vocabulary ready for form
2.27. Cache
2.27.1. Setting
Configuration > Peformance (/admin/config/development/performance)
- Always check
Cache pages for anonymous userswhich iscache- External cache system e.g. Varnish may still cache the page even when this is disabled
- Check
Cache blockswhich isblock_cachefor logged-in users. Minimum Cache Lifetimewhich iscache_lifetime- If you're not sure, better leave it as 0
- It applies to all pages and cache objects
- The value is
It's ok to serve cache objects which are stale for this value amount of time - The cache clearing is triggered by events e.g. Drupal Cron. If these events are not triggered, the cache will not be regenerated even if the expiration has passed
- e.g. If it's set to 5 mins and a new post is created, the new post appears on the homepage at least after 5 mins and Drupal Cron is run
cache_set()with no$expiretime will takecache_lifetime. Set$expireincache_setto overide.- When
cache_lifetimeis reached in Redis, cache key and value will be removed from Redis.
- Start with 15 mins and go up as you need for
Expiration of cached pageswhich ispage_cache_maximum_age- It has nothing to do with cache objects or Drupal Database
- It sets an HTTP response header
Cache-Control: public, max-age=<...>. Refer to header:cache-control:public max-age
Bandwidth Optimization:
Compress cached pages- If your server (Pantheon) already delivers content in gzip, then don't check it
- (no term)
- Always check
Aggregate and compress CSS filesandAggregate JavaScript files
External cache system may include Varnish and other proxy servers.
2.27.2. Same page request
function lili_cache($field, $set = NULL) { $custom_cache = &drupal_static(__FUNCTION__); if (!isset($custom_cache)) { $custom_cache = array(); } if ($set !== NULL && $field !== NULL ) { // $set_field_value = lili_cache('field_name', 123); $custom_cache[$field] = $set; } elseif ($field !== NULL && $set == NULL && ( !isset($custom_cache[$field]) || $custom_cache[$field] == NULL ) ) { // Set default // $set_field_value = lili_cache('field_name'); switch ($field) { case "field_name": // complicated $custom_cache[$field] = ''; break; case "truck_makes_array": $_terms = lili_cache('truck_makes_obj'); $a = []; foreach ($_terms as $k => $v) { $a[$v->name] = (tid) $v->tid; } $custom_cache[$field] = $a; break; case "truck_makes_obj": $vocab = taxonomy_vocabulary_machine_name_load('truck_makes'); $terms = taxonomy_get_tree($vocab->vid); $custom_cache[$field] = (!empty($terms)) ? $terms : []; break; } } if (!isset($custom_cache[$field])) { $custom_cache[$field] = NULL; } return $custom_cache[$field]; } // Set lili_cache('current_node',$node); // Get lili_cache('current_node');
2.27.3. Cache variables in cache table, and clear them
function my_module_function() { $my_data = &drupal_static( __FUNCTION__ ); if ( ! isset( $my_data ) ) { if ( $cache = cache_get( 'my_module_data' ) ) { $my_data = $cache->data; } else { // Do your expensive calculations here, and populate $my_data // with the correct stuff.. cache_set( 'my_module_data', $my_data, 'cache' ); // Set cache with an expiry time // cache_set('my_module_data', $my_data, 'cache', time() + 360); } } return $my_data; }
- Clear one variable cache
cache_clear_all('my_module_data', 'cache');- Clear all variables start with
my_module cache_clear_all('my_module', 'cache', TRUE);
2.27.4. Cache block d7:cache block
If there're modules setup node_grants, variable block_cache in Performance will be FALSE and disabled. You will see table cache_block is empty. In Pantheon, cache is in Redis Here's how to Enable caching for specific blocks only. In settings.php
$conf['block_cache_bypass_node_grants'] = "TRUE"; $conf['block_cache'] = "TRUE";
Set all blocks to DRUPAL_NO_CACHE (_block_get_cache_id($block)) Set the blocks to cache to have any one of
- DRUPAL_CACHE_GLOBAL
- DRUPAL_CACHE_PER_PAGE
- DRUPAL_CACHE_PER_ROLE
- DRUPAL_CACHE_PER_USER
- DRUPAL_CACHE_CUSTOM (in this case you need to setup own caching in hook_block_view or hook_block_view_MODULE_DELTA)
But not DRUPAL_NO_CACHE
2.28. File
Read local or remote file to an object of strings
$result = drupal_http_request($url); /* obj( code // int 200 headers ) */ $path = 'public://afolder/'; $replace = FILE_EXISTS_RENAME; // Default // FILE_EXISTS_REPLACE, FILE_EXISTS_ERROR file_unmanaged_save_data($result->data, $path, $replace);
file_unmanaged_save_data($data, $destination, $replace)
// $destination is folder path not folder path + filename $temp_name = 'temporary://auniquefilename' file_put_contents($temp_name,$data); // return number of bytes or false file_unmanaged_move($temp_name,$destination, $replace); // rename
Drupal download file with real file extension
/** * Download external file to temporary filesystem * with correct file extension based on content-type in http response header * * @param $url string Full external URL of a file * * @return string Local file path with correct file extension */ function _lili_save_file_with_extension( $url ) { $filename = $url; $extension = ''; $result = drupal_http_request( $url ); if ( $result->code == 200 && isset( $result->headers['content-type'] ) ) { $mime_type_extension = array( 'image/jpeg' => 'jpg', 'image/jpg' => 'jpg', 'image/png' => 'png', 'image/x-png' => 'png', 'image/gif' => 'gif' ); foreach ( $mime_type_extension as $k => $v ) { if ( strcasecmp( $k, $result->headers['content-type'] ) == 0 ) { $extension = $v; break; } } if ( $extension !== '' ) { // Save file with correct file extension $filename = 'temporary://import_' . str_replace( ".", "_", microtime( TRUE ) ) . "." . $extension; file_put_contents( $filename, $result->data ); } } return $filename; }
2.29. File Permissions d7:file permissions
sudo ./fix_drupal_permissions.sh --drupal_path=/var/www/a.ca/public --drupal_user=www-data
#!/bin/bash # Help menu print_help() { cat <<-HELP This script is used to fix permissions of a Drupal installation you need to provide the following arguments: 1) Path to your Drupal installation. 2) Username of the user that you want to give files/directories ownership. 3) HTTPD group name (defaults to www-data for Apache). Usage: (sudo) bash ${0##*/} --drupal_path=PATH --drupal_user=USER --httpd_group=GROUP Example: (sudo) bash ${0##*/} --drupal_path=/usr/local/apache2/htdocs --drupal_user=john --httpd_group=www-data HELP exit 0 } if [ $(id -u) != 0 ]; then printf "**************************************\n" printf "* Error: You must run this with sudo or root*\n" printf "**************************************\n" print_help exit 1 fi drupal_path=${1%/} drupal_user=${2} httpd_group="${3:-www-data}" # Parse Command Line Arguments while [ "$#" -gt 0 ]; do case "$1" in --drupal_path=*) drupal_path="${1#*=}" ;; --drupal_user=*) drupal_user="${1#*=}" ;; --httpd_group=*) httpd_group="${1#*=}" ;; --help) print_help;; *) printf "***********************************************************\n" printf "* Error: Invalid argument, run --help for valid arguments. *\n" printf "***********************************************************\n" exit 1 esac shift done if [ -z "${drupal_path}" ] || [ ! -d "${drupal_path}/sites" ] || [ ! -f "${drupal_path}/core/modules/system/system.module" ] && [ ! -f "${drupal_path}/modules/system/system.module" ]; then printf "*********************************************\n" printf "* Error: Please provide a valid Drupal path. *\n" printf "*********************************************\n" print_help exit 1 fi if [ -z "${drupal_user}" ] || [[ $(id -un "${drupal_user}" 2> /dev/null) != "${drupal_user}" ]]; then printf "*************************************\n" printf "* Error: Please provide a valid user. *\n" printf "*************************************\n" print_help exit 1 fi cd $drupal_path printf "Changing ownership of all contents of "${drupal_path}":\n user => "${drupal_user}" \t group => "${httpd_group}"\n" chown -R ${drupal_user}:${httpd_group} . printf "Changing permissions of all directories inside "${drupal_path}" to "rwxr-x---"...\n" find . -type d -exec chmod u=rwx,g=rx,o= '{}' \; printf "Changing permissions of all files inside "${drupal_path}" to "rw-r-----"...\n" find . -type f -exec chmod u=rw,g=r,o= '{}' \; printf "Changing permissions of "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n" cd sites find . -type d -name files -exec chmod ug=rwx,o= '{}' \; printf "Changing permissions of all files inside all "files" directories in "${drupal_path}/sites" to "rw-rw----"...\n" printf "Changing permissions of all directories inside all "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n" for x in ./*/files; do find ${x} -type d -exec chmod ug=rwx,o= '{}' \; find ${x} -type f -exec chmod ug=rw,o= '{}' \; done echo "Done setting proper permissions on files and directories"
2.30. Image Handling (Core)
2.30.1. Setting
Define Image Styles /admin/config/media/image-style
Use it in image field display or in code.
Image toolkit /admin/config/media/image-toolkit reduces the JPEG quality if Image Styles is applied.
Both Image Styles and Image toolkit don't modify the original images but instead create new images in real time.
2.30.2. Code
An image is a file. Get a image field:
$w = entity_metadata_wrapper('node', $node);
$images = $w->field_images->value();
if (isset($images)) {
// If the image field is set to only hold one image, then
// $images is ['uri'=>'...']
foreach ($r as $k => $v) {
$r[$k]['url'] = file_create_url($v['uri']);
// convert internal URI to public URL. Drupal doens't provide public url by default.
// This step doesn't create a file, just create a link
// Get the image style public url
$r[$k]['thumbnail'] = image_style_url('thumbnail', $v['uri']);
// It creates an image styled link in db, when requesting the public url, the image will be created
}
}
2.30.3. Get File URI from db d7:db:file uri
2.31. Basic Ajax
// Create a menu to receive ajax $items['newsletter/builder_entry/%'] = array( 'title' => "", 'page callback' => '_lili_add_session_handler', 'page arguments' => array(2), 'access callback' => 'user_access', 'access arguments' => array('newsletter builder access'), 'type' => MENU_CALLBACK, ); function _lili_add_session_handler($command) { if (!isset($_POST['ids'])) { return drupal_not_found(); } if ($command === "manage") { // Add to the session variable detailing what items are added $_SESSION['session_name'] = array(); $ids = json_decode($_POST['ids']); foreach ($ids as $id) { $id = trim($id); $_SESSION['session_name']["item_" . $id] = $id; } } else { return drupal_not_found(); } }
$.ajax({
type: "POST",
url: '?q=newsletter/builder_entry/manage',
data: "ids=" + 'some string',
success: function(e) {
// do something
},
dataType: "json"
});
2.32. Javascript API
{ 'settings': {}, 'behaviors': {}, 'themes': {}, 'locale': {} }
2.32.1. Settings
Pass PHP variable to javascript
PHP. Use camelCasing
$my_settings = array( 'basePath' => $base_path, 'animationEffect' => variable_get('effect', 'none') ); drupal_add_js(array('myModule'=>$my_settings), 'setting');
Javascript
var basePath = Drupal.settings.myModule.basePath; var effect = Drupal.settings.myModule.animationEffect;
2.32.2. Behaviors
https://www.lullabot.com/articles/understanding-javascript-behaviors-in-drupal https://www.amazeelabs.com/en/blog/drupal-behaviors-quick-how https://www.drupal.org/node/756722
Any property defined in Drupal.behaviors will get called when DOM is ready or when Drupal.attachBehaviors() runs. Drupal.attachBehaviors(); means attach all behaviors. When DOM is ready, Drupal runs:
// Attach all behaviors. $(function () { Drupal.attachBehaviors(document, Drupal.settings); }); // Syntax: Drupal.attachBehanviors(context, settings); // When context is not defined, it's the document object in javascript // when settings is not defined, Drupal.settings is used // So Drupal.attachBehaviors(); is the same as Drupal.attachBehaviors(document, Drupal.settings);
Other modules might run multiple Drupal.attachBehaviors(); or Drupal.attachBehaviors(custom_context, custom_settings);
All properties defined in Drupal.behaviors will get called when Drupa.attachBehaviors(…, …); runs! Some common situations:
- After an admin overlay has been loaded into the page.
- After AJAX Form API has submitted a form
- When an AJAX request returns a command that modifies HTML, such as ajax_command_replace()
- CTools calls it after a modal has been loaded
- Media calls it after media browser has been loaded
- Panels calls it after in-place editing has been completed
- Views calls it after loading a new page that uses AJAX
- Views Load More calls it after loading the next chunk of items
- Javascript from custom modules may call Drupal.attahcBehaviors() when they add or change parts of the page.
- Such as drupal_add_js('jQuery(document).ready(function () { Drupal.attachBehaviors(); });', 'inline');
So you need to attach any event once using $('',context).once() !
This is mymod_path/js/drupal.behaviors.js
(function ($) { Drupal.behaviors.MODULENAME = { attach: function (context, settings) { console.log('This runs every time Drupal.attachBehaviors(); is called'); // Run something only once (e.g. set jQuery event) $('.nav-flyout', context).once('remove-modals', function () { // After this is run, all elements selected will have a new class 'remove-modals-processed' $(document).keyup(function (e) { if (e.keyCode == 27) { $('.nav-flyout', context).removeClass('js-flyout-active'); } }); }); } } }(jQuery));
In Drupal
function hook_preprocess_page(&$vars) { // Append a property in Drupal.settings only. And before a behavior is attached and it will be run e.g. when DOM is ready drupal_add_js(array('MODULENAME' => array('php_var_1' => variable_get('php_var_1', ''))), 'setting'); drupal_add_js(drupal_get_path('module', 'MODULENAME') . '/js/drupal.behaviors.js'); }
2.33. Email
Define an email template using hook_mail()
function lili_mail($key, &$message, $params) { // Need to overwrite headers content type if email is html $message['headers']['Content-Type'] = 'text/html; charset=UTF-8; format=flowed'; // $message['from'] is initially set by $from from drupal_mail(). // If not provided, will use sidewide setting // $message['to'] and 'language' are set by $to from drupal_mail(). switch ($key) { case 'template_1': // Use $params $message['subject'] = 'This is subject'; $message['body'][] = 'Extra'; $message['headers']['Bcc'] = 'bcc@abc.com'; break; } } // Send email $module = 'lili'; // module name that the template is defined hook_mail $to = 'to@abc.com'; $from = 'from@abc.com'; // Default Null $language = language_default(); $send = TRUE; // Default TRUE $params = array(); // Extra params to pass to hook_mail() as $params // hook_mail_alter can change to false so that it doesn't get sent out $result = drupal_mail($module, 'template_1', $to, $language, $params, $from, $send); /* $result => array( 'result' => NULL (cancelled by hook_mail_alter()) or $system->mail($message) (!$result['result'] is true means fail) ) */
2.34. Life Cycle
includes/bootstrap.inc- Global variables and functions
- abstract class DrupalCacheArray
- class SchemaCache extends DrupalCacheArray
drupal_bootstrap( DRUPAL_BOOTSTRAP_FULL );DRUPAL_BOOTSTRAP_CONFIGURATION_drupal_bootstrap_configuration();
includes/file.phar.incdrupal_settings_initialize();
conf_path() . '/settings.php';- If
sites/sites.phpexists, load it'<port>.<domain>.<path>' => 'directory'e.g.- for alias/site
8080.www.drupal.org.mysite.test, confg path issites/example.com - for alias/site
a.com, config path issites/a - Try to match alias with
$_SERVER['SCRIPT_NAME']and$_SERVER['HTTP_HOST']
- If no
sites/sites.phpor no match, returnsites/default
includes/request-sanitizer.inc- class DrupalRequestSanitizer
- (no term)
DrupalRequestSanitizer::sanitize();
- If
DRUPAL_BOOTSTRAP_PAGE_CACHE_drupal_bootstrap_page_cache();- Attempts to serve a page from the cache
includes/cache.inc- functions, class DrupalCacheInterface, DrupalDatabaseCache implements DrupalCacheInterface
$conf['cache_backends']- include each cache_backend
drupal_bootstrap(DRUPAL_BOOTSTRAP_VARIABLES, FALSE);- bootstrap variables
- (no term)
- If there is no session cookie and cache is enabled (or forced), try to serve a cached page
DRUPAL_BOOTSTRAP_DATABASE_drupal_bootstrap_database();- Initializes the database system and registers autoload functions
includes/database/database.inc
DRUPAL_BOOTSTRAP_VARIABLES_drupal_bootstrap_variables();- Loads system variables and all enabled bootstrap modules
variable_get('lock_inc', 'includes/lock.inc')- functions
includes/module.inc- functions
module_load_all(TRUE);- Loads all the modules that have been enabled in the system table
foreach (module_list(TRUE, $bootstrap) as $module) { drupal_load('module', $module); }- load bootstrap modules only (for returning cache content)
- (no term)
- Sanitize
$_GET['destination']and$_REQUEST['destination']
DRUPAL_BOOTSTRAP_SESSIONvariable_get('session_inc', 'includes/session.inc')- functions
drupal_session_initialize()- Initializes the session handler, starting a session if needed
DRUPAL_BOOTSTRAP_PAGE_HEADER_drupal_bootstrap_page_header();- Invokes hook_boot(), initializes locking system, and sends HTTP headers
bootstrap_invoke_all('boot');- Invokes a bootstrap hook in all bootstrap modules that implement it
DRUPAL_BOOTSTRAP_LANGUAGEdrupal_language_initialize();- Initializes all the defined language types
DRUPAL_BOOTSTRAP_FULLincludes/common.inc- Global variables and functions
- (no term)
_drupal_bootstrap_full();
variable_get('path_inc', 'includes/path.inc');- functions
includes/theme.inc- global variables, class ThemeRegistry extends DrupalCacheArray and functions
includes/pager.inc- class PagerDefault extends SelectQueryExtender and functions
variable_get('menu_inc', 'includes/menu.inc')- global variables and functions
includes/tablesort.inc- class TableSort extends SelectQueryExtender and functions
includes/file.inc- global variables and functions
includes/stream_wrappers.inc- global variables and classes
includes/unicode.inc- global variables and functions
includes/image.inc- functions
includes/form.inc- functions
includes/mail.inc- global variable, class MailSystemInterface and functions
includes/actions.inc- functions
includes/ajax.inc- functions
includes/token.inc- functions
includes/errors.inc- functions
module_load_all();- load
system_list('module_enabled')modules drupal_path_initialize();- Initialize the $_GET['q'] variable to the proper normal path
menu_set_custom_theme();drupal_theme_initialize();module_invoke_all('init');- Invokes a hook_init in all enabled modules
2.35. Hook System d7:hook system
- Name
- Core
- page, node, taxonomy_term
- Invoke
drupal_alter($type, &$data, &$context = NULL, &$context2 = NULL, &$context3 = NULL)Pass alterable variables to specific
hook_TYPE_alter()implementations- Max 2 alterable arguments is supported (
$context3is not supported anymore) - In case more arguments need to be passed and alterable, use
$context2and leave$context3undefined
$context = array( 'alterable' => &$alterable, 'unalterable' => $unalterable, 'foo' => 'bar', ); drupal_alter('mymodule_data', $alterable1, $alterable2, $context); // note that if any object are passed in `$context`, it will be passed by reference in PHP5 // If it's required that there's no implementation alters a passed object in $context, clone the object and pass $context = array( 'unalterable_object' => clone $object, ); drupal_alter('mymodule_data', $data, $context);
- $type
- string or arry e.g.
'form''links''node_content'. If it's array,hook_TYPE_alter()is invoked for each value in array, ordered first by module, and then for each module, in the order of values in$type. e.g. Form API['form', 'form_'.$fomr_id]to executehook_form_alter()and thenhook_form_FORM_ID_alter() - $data
- to be altered
- Max 2 alterable arguments is supported (
module_invoke_all($hook)- Invoke a hook in all enabled modules. Variables are passed by value not by reference
- $hook
- hook name.
call_user_func_array($module . '_' . $hook, $args) - $args
- Arguments to pass to the hook
module_invoke($module, $hook)- invokes a hook in a particular module. All arguments are passed by value
- Hook order
- Module weight
default 0. The lower the higher priority. Can be negative numbers
- Check all module weights
print implode(PHP_EOL, module_list());- SQL update
UPDATE system SET weight=999 WHERE name = 'mymodule'- Per hook
hook_module_implements_alter(&$implementations, $hook). Hook is invoked duringmodule_implements()
- $implementations
- Array keyed by the module's name. The value of each item corresponds to a $group, which is usually FALSE, unless the implementation is in a file names $module.$group.inc
- $hook
The name of the module hook being implemented
unction hook_module_implements_alter(&$implementations, $hook) { if ($hook == 'rdf_mapping') { // Move my_module_rdf_mapping() to the end of the list. module_implements() // iterates through $implementations with a foreach loop which PHP iterates // in the order that the items were added, so to move an item to the end of // the array, we remove it and then add it. $group = $implementations['my_module']; unset($implementations['my_module']); $implementations['my_module'] = $group; } } // always execute first for hook_form_alter and form_FORM_ID_alter function my_module_module_implements_alter(&$implementations, $hook) { if (($hook == 'form_FORM_ID_alter' || $hook == 'form_alter') && isset($implementations['module_name']) { $module = 'my_module'; $group = array($module => $implementations[$module]); unset($implementations[$module]); $implementations = $group + $implementations; } if ($hook != 'hook_form_alter') { return; } }
- module
.installfile - d7:hook_install
- Use contributed modules
module_weightandUtility
- module
- (no term)
- Module name alphabetical order
2.36. OOP
- D7 refers to oop_examples module
- One quick way to include a class is to put the class inside any file e.g.
$moddir/lib/reader.incand include when it's needed usingmodule_load_include('inc', 'modename', 'lib/reader');
3. WordPress
3.1. Core Update
- https://codex.wordpress.org/WordPress_Versions
- https://codex.wordpress.org/Configuring_Automatic_Background_Updates
wp-includes/version.phporreadme.htmlwp-includes/version.phpalso has- wp db version
- required php version
- required mysql version
- local package (language)
- tinymce version
define( 'WP_AUTO_UPDATE_CORE', true );- Value of true – Development, minor, and major updates are all enabled
- Value of false – Development, minor, and major updates are all disabled
- Value of 'minor' – Minor updates are enabled, development, and major updates are disabled
Fine tune
add_filter( 'allow_dev_auto_core_updates', '__return_true' ); // Enable development updates add_filter( 'allow_minor_auto_core_updates', '__return_true' ); // Enable minor updates add_filter( 'allow_major_auto_core_updates', '__return_true' ); // Enable major updates
3.1.1. Manual Update Core
- https://codex.wordpress.org/Upgrading_WordPress
- https://codex.wordpress.org/Upgrading_WordPress_Extended
- Backup
- Disable all plugins
- Delete old wp-includes and wp-admin and replace with new ones
- Go to new wp-content and copy and replace existing files and directories
- Replace files at root directory
- Visit /wp-admin and it will give you a link
- http://example.com/wp-admin/upgrade.php
- Update Permalinks and .htaccess from wp-admin UI
- Install updated Plugins and Themes
- Reactivate plugins
3.1.2. Release
See if wp-config.php .htaccess files are modified 3.8.6 > 4.5.10 > (db) 4.9.1 > 4.9.4
3.1.3. TS: update-core.php blank
- Add this to disable WP to check FTP
define('FS_METHOD', 'direct');- (no term)
- Refer to wp:ftp https://stackoverflow.com/questions/28598547/wordpress-core-update-fail-4-1-to-4-1-1
3.2. Migrate database
At your previous host, access phpMyAdmin. Load your WordPress database by clicking the database name in the left hand menu pane. Click the "export" tab at the top right. Under "export, highlight all the tables and select "SQL". These are the default settings. Under "options" make sure that "add DROP TABLE/ VIEW/ PROCEDURE/ FUNCTION" is selected. Ensure that "Save as File" towards the bottom of the page is selected. Click Go
For import, just hit Import tab and import. Watch out for the timeout limit.
Change website url Open wp_options table, change siteurl to http://yourwebsite.com
Do it on code level
define('WP_HOME','http://'.$_SERVER[HTTP_HOST].'/'); define('WP_SITEURL','http://'.$_SERVER[HTTP_HOST].'/');
Go to wp-admin > Settings > Permalink and save to update the permalink
3.3. Redirect a path a Wordpress website wp:redirect path to wp
abc.com/enterprise to another Wordpress website abc.com DB is abc_enterprise and abc
abc.com is inside a docker container using apache2
abc.com uses Nginx and initial setup is
server { listen 80; server_name abc.com www.abc.com; location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $http_host; proxy_pass http://127.0.0.1:9977; } }
3.3.1. Method 1: Apache
Directly create or copy a sub directory under abc.com root /var/www/html/enterprise
Create a new database called abc_enterprise and import the db /var/www/html/enterprise/wp-config.php
If the enterprise site is a complete new wp installation, you don't need to do this
define( 'WP_HOME', '/enterprise' ); define( 'WP_SITEURL', '/enterprise' ); // for localhost // define('WP_HOME','http://'.$_SERVER[HTTP_HOST].'/enterprise'); // define('WP_SITEURL','http://'.$_SERVER[HTTP_HOST].'/enterprise'); // Change the db setting define('DB_NAME', 'abc_enterprise');
var/www/html/enterprise.htaccess
# BEGIN WordPress <IfModule mod_rewrite.c> RewriteEngine On RewriteBase /enterprise/ RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /enterprise/index.php [L] </IfModule> # END WordPress
Then go to wp-admin and save the permalink
abc.com/enterprise/abc has $_SERVER['REQUEST_URI'] appear as /abc on /var/www/html/enterprise/index.php
- Redirect abc.com/xyz to abc.com/enterprise/ijk use var/www/html.htaccess
RedirectMatch 302 ^/psup/?$ /enterprise/ijk/
- Rewrite abc.com/ijk to abc.com/enterprise/real-page so that URL remains abc.com/ijk in address bar
use var/www/html.htaccess
<IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^ijk/?(.*)$ /enterprise/ijk [L] </IfModule>
Then in /var/www/html/enterprise/wp-content/themes/your-theme/functions.php
NOTE: $_SERVER['REQUEST_URI'] is /ijk in ./enterprise/index.php
function cs_rewrite_request($query){ $request_uri = urldecode($_SERVER['REQUEST_URI']); //var_dump($request_uri); /* for pages */ if( 0 === strpos($request_uri, '/ijk') ){ $query['pagename'] = urlencode('real-page'); unset($query['name']); } return $query; } add_filter( 'request', 'cs_rewrite_request', 9999, 1 );
If you want abc.com/enterprise/real-page to redirect to abc.com/ijk which shows the real-page var/www/html.htaccess
RedirectMatch 302 ^/enterprise/real-page/?$ /ijk/
Fix image urls
UPDATE wp_posts SET post_content=(REPLACE (post_content, '<old url>','<new url>')); UPDATE wp_posts SET post_content=(REPLACE (post_content, 'abc-ent-dev.com','abc.com/enterprise'));
3.3.2. Method 2: (Not working..) Nginx proxy to another
Refer to nginx:redirect path to proxy
abc.com Nginx becomes
server { listen 80; server_name alectraconservation.com www.alectraconservation.com; location ^~ /enterprise/ { proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $http_host; proxy_pass http://127.0.0.1:8787/; } location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $http_host; proxy_pass http://127.0.0.1:9977; } }
Refer to http://www.ur-ban.com/2015/07/27/nginx-proxy_pass-wordpress-in-a-sub-directory/ Change WP_HOME and WP_SITEURL
$_SERVER['REQUEST_URI'] = str_replace("/wp-admin/", "/enterprise/wp-admin/", $_SERVER['REQUEST_URI']); define( 'WP_SITEURL', '/enterprise' ); define( 'WP_HOME', '/enterprise' );
3.4. Load WP Core in sub directory under same domain
sub directory has a customized PHP application and it needs to use WP Core to determine if a user is logged in
https://www.webhostinghero.com/wordpress-authentication-integration-with-php/ http://dovy.io/wordpress/authenticating-outside-of-wordpress-on-diff-domain/
subfolder/index.php
define( 'WP_USE_THEMES', false ); // NOT load the theme at all or any of that content. define( 'COOKIE_DOMAIN', false ); // NOT verify the cookie domain. We just want to take the cookie as it is. define( 'DISABLE_WP_CRON', true ); // Optional // load less core files. Good for wp REST API request. e.g. global $wp, $wp_query, $wp_the_query are set to NULL // define( 'SHORTINIT', TRUE ); // if(!session_id()) session_start(); // I find this not necessary require_once(__DIR__ . "/../wp-load.php"); // You will find some errors, fix errors in your custom themes and plugins // clear cache and try again // see wp:user
3.5. Global Variables
3.5.1. $wp_version
global $wp_version; if ( version_compare( $GLOBALS['wp_version'], '4.7-alpha', '<' ) ) { require get_template_directory() . '/inc/back-compat.php'; return; }
3.5.2. $_GET, $_POST, $_COOKIE, $_SERVER
WP first detects if Magic Quotes, if so, remove them using stripslashes_deep (wp function) and then add slashes using add_magic_quotes which uses php:addslashes function. In order to get values, use stripslashes_deep again
$value = stripslashes_deep($_POST['name']); // or simply $myPost = stripslashes_deep($_POST); // in custom PHP app, detect if WP already defines the function and later remove the slashes $myPost = $_POST; if (function_exists('stripslashes_deep')) { // This means WP has already added slashes to $_POST. Remove slashes $myPost = stripslashes_deep($myPost); }
3.5.3. $wp_query $GLOBALS['wp_query'] wp:global:wp_query
Refer to WP_Query
$p = $GLOBALS['wp_query']; // first post $p->posts[0] // total number of posts found by query echo $p->found_posts; // number of posts being displayed echo $p->post_count;
3.5.4. $wp_rewrite wp:global:rewrite
- Refer to wp:options:rewrite_rules
3.5.5. $post wp:global:post
https://codex.wordpress.org/Class_Reference/WP_Post
global $post; $post_slug = $post->post_name;
3.5.6. $wpdb wp:global:wpdb
- https://codex.wordpress.org/Class_Reference/wpdb
$wpdbprovides an interface which WP_Query uses to query- wp:plugin:w3-total-cache
- If wp core doesn't support
msql_connectbutmysqliand the environment doesn't have msql extension (since PHP 7.0), then replacewp-includes/wp-db.phpwith the latest wp version without upgrading wp core. Although upgrading wp core is recommended
3.5.6.1. Use $wpdb to create raw query
https://code.tutsplus.com/tutorials/writing-custom-queries-in-wordpress--wp-25510
global $wpdb; $query = " SELECT * FROM wp_terms wt INNER JOIN wp_term_taxonomy wtt ON wt.term_id = wtt.term_id WHERE wtt.taxonomy = 'post_tag' AND wtt.count = 0"; $wpdb->query($query); // int number of rows affected/selected and false when there is an error // array. $wpdb->get_results($query, ARRAY_A); // get one variable. Useful when query result has only one column $wpdb->get_var($query); // e.g. $user_count = $wpdb->get_var( "SELECT COUNT(*) FROM $wpdb->users" ); echo "<p>User count is {$user_count}</p>"; // return one row. result_type can be OBJECT, ARRAY_A or ARRAY_N (object, associative array or numbered array). Offset is an integer with a default of 0 $wpdb->get_row($query, ARRAY_A, 3); // get a column. Output will be a dimensional array. Empty array if no result. The second parameter is the column offset $wpdb->get_col($query, 3); // to prevent SQL injection, use prepare $wpdb->query( $wpdb->prepare( "INSERT INTO test_table (post_id, animal, food) VALUES ( %d, %s, %s )", array( 10, 'monkey', 'apple' ) )); $wpdb->show_errors(); $wpdb->hide_errors(); $wpdb->flush(); $wpdb->insert( 'foods', array( 'fruit' => 'apple', 'year' => 2012 ), array( '%s', '%d' ) ); $wpdb->update( 'foods', array( 'fruit' => 'apple', // string 'year' => 'value2' // integer (number) ), array( 'ID' => 1 ), array( '%s', // value1 '%d' // value2 ), array( '%d' ) ); // column info $wpdb->get_col_info('type', offset); // Type: the information you want to retrieve, some examples are here // name – column name (this is the default) // table – name of the table the column belongs to // max_length – maximum length of the column // not_null – 1 if the column cannot be NULL // more can be found in the WordPress Codex WPDB reference // Offset: specify the column from which to retrieve information (0 is the first column) // reference WordPress tables $wpdb->posts; $wpdb->postmeta; $wpdb->comments; $wpdb->commentmeta; $wpdb->terms; $wpdb->term_taxonomy; $wpdb->term_relationships; $wpdb->users; $wpdb->usermeta; $wpdb->links; $wpdb->options; // e.g. $user_count = $wpdb->get_var( "SELECT COUNT(*) FROM $wpdb->users" );
3.5.7. $current_user wp:global:current_user
3.5.8. $wp_filter, $wp_current_filter wp:global:wp_filter wp:global:wp_current_filter
Use current_action() or current_filter() to return the current filter
3.5.9. $template wp:global:template
Which php file is used for template
global $template; print_r( $template );
3.5.10. Image Sizes $_wp_additional_image_sizes
Refer to wp:debug:image sizes
3.5.11. Post statuses $wp_post_statuses wp:global:wp_post_statuses
3.5.12. $wp_customize wp:global:wp_customize
3.6. wp-config.php
3.6.1. wp:wp-config:DISALLOW_FILE_EDIT
define( 'DISALLOW_FILE_EDIT', true );
3.7. Options, Settings wp:options
/wp-admin/options.php- show all options. Serialized data is not shown
- (no term)
- Option Reference
- stylesheet
- The slug of the currently activated stylesheet (style.css). Which is the active theme's dir name. Also called theme slug wp:options:stylesheet
theme_mods_{$theme_slug}- wp:options:theme_modstheme_slug
- Refer theme_slug in above wp:options:stylesheet
- sticky_posts
- array of post ID's (only post type post)
- permalink_structure
- e.g.
/%postname%/ - rewrite_rules
- serialized wp:options:rewrite_rules
- Custom post type
[ "articles/(.+?)(?:/([0-9]+))?/?$" => "index.php?articles=$matches[1]&page=$matches[2]" ]- Custom taxonomy
[ "neckline/([^/]+)/?$" => "index.php?gallery_neckline=$matches[1]" ]
- noindex, nofollow
- Settings > Reading > toggle
Search Engine Visibility - Discourage search engines from indexing this site
3.8. Email PHPMailer
Add this after require_once(ABSPATH . 'wp-settings.php'); in wp-config.php
This way you don't need to have linux:sendmail installed on Linux
// configure phpmailer add_action( 'phpmailer_init', 'mail_relay' ); function mail_relay( $phpmailer ) { $phpmailer->isSMTP(); $phpmailer->Host = 'smtp.gmail.com'; $phpmailer->SMTPAutoTLS = true; $phpmailer->SMTPAuth = true; $phpmailer->Port = 465; $phpmailer->Username = '<admin email>'; $phpmailer->Password = '<app password>'; // Additional settings $phpmailer->SMTPSecure = "ssl"; $phpmailer->From = "<admin email>"; $phpmailer->FromName = "<your name>"; } // configure phpmailer.
Without bootstrap wp, assuming test.php file is at root
error_reporting(E_ALL); ini_set('display_errors', 1); echo 'hello'; // PHPMailer Gmail Example // https://github.com/PHPMailer/PHPMailer/blob/master/examples/gmail.phps $recipient_email = 'a@b.com'; $recipient_name = 'First Last'; $sender_name = 'Sender Name'; $subject = 'Testing'; $html = (' <html> <head> <title>This is a test</title> </head> <body> <h1>Testing headline</h1> <p>Testing body content</p> </body> </html> '); $text = 'pure text testing'; // php's native mail // mail($recipient_email, $subject, $text); require("wp-includes/class-phpmailer.php"); $mail = new PHPMailer(); $mail->isSMTP(); // Set mailer to use SMTP //Enable SMTP debugging // 0 = off (for production use) // 1 = client messages // 2 = client and server messages $mail->SMTPDebug = 2; $mail->Host = 'smtp.gmail.com'; // use // $mail->Host = gethostbyname('smtp.gmail.com'); // if your network does not support SMTP over IPv6 $mail->SMTPAuth = true; // set to false if Username, Password, SMTPSecure are empty $mail->Username = 'username@gmail.com'; $mail->Password = 'password'; $mail->SMTPSecure = 'tls'; // Enable TLS encryption, `ssl` also accepted $mail->Port = 587; // 587 for tls $mail->CharSet = 'UTF-8'; $mail->SetFrom('donotreply@b.com', $sender_name); $mail->AddAddress($recipient_email, $recipient_name); $mail->Subject = $subject; //$mail->addReplyTo('replyto@example.com', 'First Last'); //Replace the plain text body with one created manually // $mail->AltBody = 'This is a plain-text message body'; //Attach an image file //$mail->addAttachment('images/phpmailer_mini.png'); $mail->MsgHTML($html); if (!$mail->Send()) { echo "Mailer Error: " . $mail->ErrorInfo; } else { echo "successful!"; }
3.9. PHP redirect wp:php:redirect
- https://rudrastyh.com/wordpress/change-specific-urls.html
- Use functions.php to achieve
Redirect, same can be achieved using .htaccess
function rudr_url_redirects() { /* in this array: old URLs=>new URLs */ $redirect_rules = array( array('old'=>'/category/uncategorized/','new'=>'/category/Uncategorized/'), // category array('old'=>'/contacts/','new'=>'/Contacts/'), // page array('old'=>'/hello-world/','new'=>'/hello-planet/'), // post array('old'=>'/tag/wordpress/','new'=>'/tag/WordPress/') // post tag ); foreach( $redirect_rules as $rule ) : // if URL of request matches with the one from the array, then redirect if( urldecode($_SERVER['REQUEST_URI']) == $rule['old'] ) : wp_redirect( site_url( $rule['new'] ), 301 ); exit(); endif; endforeach; } add_action('template_redirect', 'rudr_url_redirects');
Redirect by changing request based on the $_SERVER['REQUEST_URI']
function rudr_rewrite_request($query){ $request_uri = urldecode($_SERVER['REQUEST_URI']); /* for categories */ if ( $request_uri == '/category/Uncategorized/' ) $query['category_name'] = 'uncategorized'; /* for pages */ if ( $request_uri == '/Contacts/' ){ $query['pagename'] = urlencode('contacts'); unset($query['name']); } /* for posts */ if ( $request_uri == '/hello-planet/' ) $query['name'] = 'hello-world'; /* for tags */ if( $request_uri == '/tag/WordPress/' ) $query['tag'] = 'wordpress'; return $query; } add_filter( 'request', 'rudr_rewrite_request', 9999, 1 );
Change permalink For posts and pages
function rudr_post_permalink( $url, $post ){ if( !is_object( $post ) ) $post = get_post( $post_id ); $replace = $post->post_name; /* We should use a post ID to make a replacement. It is required if you use urf-8 characters in your URLs */ if( $post->ID == 1 ) $replace = 'hello-planet'; if( $post->ID == 12 ) $replace = 'Contacts'; $url = str_replace($post->post_name, $replace, $url ); return $url; } add_filter( 'post_link', 'rudr_post_permalink', 'edit_files', 2 ); add_filter( 'page_link', 'rudr_post_permalink', 'edit_files', 2 ); add_filter( 'post_type_link', 'rudr_post_permalink', 'edit_files', 2 );
For categories and tags
function rudr_term_permalink( $url, $term, $taxonomy ){ $replace = $term->slug; /* by ID as well */ if( $term->term_id == 5 ) $replace = 'Uncategorized'; if( $term->term_id == 55 ) $replace = 'WordPress'; $url = str_replace($term->slug, $replace, $url ); return $url; } add_filter( 'term_link', 'rudr_term_permalink', 10, 3 );
3.10. WP-CLI wp-cli
3.10.1. WP CLI Install, Upgrade
# Assuming you are at ~ folder curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar # Check phar php wp-cli.phar --info chmod +x wp-cli.phar # GoDaddy Shared Hosting doesn't allow sudo. Ignore this for GoDaddy # sudo mv wp-cli.phar /usr/local/bin/wp alias wp='~/wp-cli.phar' echo "alias wp='~/wp-cli.phar'" >> .bashrc source .bashrc
Set Global parameters in wp-cli.yml under WordPress root folder http://wp-cli.org/config/
Use – in front for wp command
# Very important. To avoid error: $_SERVER['HTTP_HOST'] url: http://www.yourwebsite.com
Use --skip-plugins --skip-themes to avoid errors while loading plugins/themes that cause this kind of error
Fatal error: Uncaught Error: Call to a member function xxx on null
Check wp-cli installation
wp --info, wp plugin list --debug or wp plugin list --skip-plugins --debug
If you see this error, time to upgrade WP CLI
Fatal error: Class 'WP_Post_Type' not found in /Applications/MAMP/htdocs/wp-includes/post.php on line 1031
sudo wp cli update
3.10.2. Global Parameters
- –debug
- –quiet
- –user=<id|login|email>
- Load PHP file, can be used multiple times
3.10.3. Command documentation
https://github.com/wp-cli and go to db-command/src/DB_Command.php for command db.
3.10.4. user
# List users terminus wp 'user list' --site=<site> --env=<env> terminus wp mysite.live -- user list lando terminus wp mysite.live user list wp user list # Add a user, a random password is assigned wp user create lili li@xxx.ca --role=administrator # add a user, prompt for a password wp user create lili li@xxx.ca --role=administrator # Update password wp user update lili --user_pass='lettersandnumbers' # Prompt for password with special characters wp user update lili --prompt=user_pass # reset password (change to a random one and no email will be sent) wp user reset-password adminname editorname # Give role wp user add-role lili administrator # remove role wp user remove-role lili subscriber # in case of `--` cannot be used # a random password is assigned wp user create lili li@a.ca wp user add-role lili administrator # then reset password from front end
3.10.5. core
wp core download wp core version wp core update
3.10.6. plugin
Wordpress Official Plugin Directory
# List installed plugins wp plugin list # install without activate wp plugin install fly-dynamic-image-resizer # install and activate wp plugin install custom-post-type-ui --activate wp plugin status wp plugin update akismet wp plugin activate plugin_name wp plugin deactivate plugin_name wp plugin uninstall Plugin_name lando terminus wp mysite.dev plugin install post-expirator lando terminus wp mysite.dev plugin activate post-expirator
3.10.7. package
# List installed packages wp package list # Install a package wp package install aaemnnosttv/wp-cli-login-command
3.10.8. theme
wp theme status
3.10.9. login
- user
- can be a User ID, username/login or email address.
wp package install aaemnnosttv/wp-cli-login-command # find a user wp user list # create a temp login url for a user wp login create <user> [options] # Will ask you to install a plugin which contain only one .php file wp login install --activate --allow-root # run create again # Make all exisiting magic links to fail wp login invalidate # Deactivate the plugin wp login toggle off --allow-root
Options
- –expires=<seconds>
- default 900 (15 minutes)
3.10.10. media
# regenerate by attachment IDs wp media regenerate 123 124 # regenerate all thumbnails wp media regenerate --yes # regenerate with IDs between seq 1000 2000 | xargs wp media regenerate # regenerate image size "large" for all images wp media regenerate --image_size=large
3.10.11. post wp-cli:post
3.10.11.1. post list [--<field>=<value>]... [--field=<field>] [--fields=<fields>...]... [--format=<format>]
--post_typeand other args in WP_Query can be passed- https://developer.wordpress.org/reference/classes/wp_query/
wp post list --post_type=post --fields=post_title,post_name,post_date,post_status --posts_per_page=5 wp post list --post_type=post --format=count # list attachment wp post list --post_type='attachment' # list posts by post id's wp post list --post__in=1,3 # list posts by author wp post list --author__in=1,3 # delete all posts of a post type wp post delete $(wp post list --post_type='page' --format=ids) # --force to skip trash and delte permanently # delete all posts in trash wp post delete $(wp post list --post_status=trash --format=ids) # delete all attachments by an author wp post delete $(wp post list --post_type=attachment --author__in=1,3 --format=ids) # delete revisions wp post delete $(wp post list --post_type='revision' --format=ids) # meta query wp post list --post_type=post --meta_key=mycustomfield '--meta_compare=NOT EXISTS' wp post list --post_type=post --meta_key=wpcf-abc --meta_value='abc xyz' wp post list --post_type=post --meta_key=wpcf-abc --meta_value='abc xyz' '--meta_compare=!='
3.10.11.2. post delete <id>... [--force] [--defer-term-counting]
3.10.11.3. post update <id>... [--field=value]...
- Warning!!! Must define
--post_namefor updating existing posts - Otherwise post_name is set to empty
- Better to use wp-admin Bulk Edit to trigger events or use raw query to update existing posts
- (no term)
- It calls wp:f:wp_insert_post
3.10.11.4. post meta wp-cli:post meta
- Have to find out and set the private field value
_custom_orderwhich refers to the wp:plugin:acf field setting in tablewp_postswithpost_name=field_RANDOM
# wp post meta update <id> <key> <value> wp post meta update 4045 custom_order 0 _custom_order field_59038b5c9457b for id in $(wp post list --category_name=arizona-business --fields=ID --format=ids); do wp post meta update $id custom_order 0; done for id in $(wp post list --category_name=arizona-business --fields=ID --format=ids); do wp post meta update $id _custom_order field_59038b5c9457b; done # sometimes `wp post list` returns PHP error.. copy those ids and run for id in 1234 1232 1231; do wp post meta update $id custom_order 0; done for id in 1234 1232 1231; do lando wp post meta update $id custom_order 0; done lando ssh --user=root # get a post with _custom_order field reference terminus wp mysite.li-dev -- post meta get 12345 _custom_order # get a list of posts to update custom fields terminus wp mysite.li-dev -- post list --post_type=speaker --fields=ID --format=ids --allow-root # fast terminus wp mysite.li-dev -- post meta update 123 124 custom_order 0 terminus wp mysite.li-dev -- post meta update 123 124 _custom_order field_CHANGERANDOM # very slow for id in 1234 1232 1231; do terminus wp mysite.li-dev -- post meta update $id custom_order 0; done for id in 1234 1232 1231; do terminus wp mysite.li-dev -- post meta update $id _custom_order field_CHANGERANDOM; done
3.10.12. db check
3.10.13. db search
https://developer.wordpress.org/cli/commands/db/search/
wp db search 'your search string' # regex wp db search 'https?://' --regex --stats
3.10.14. db export
https://developer.wordpress.org/cli/commands/db/export/
# Export database with drop query included $ wp db export --add-drop-table Success: Exported to 'wordpress_dbase-db72bb5.sql'. wp db export --add-drop-table /var/www/html/devops/db/dump.sql
3.10.15. db size
wp db size --tables --size_format=mb
3.10.16. db query
wp db query 'SELECT id FROM wp_posts WHERE post_content LIKE "%trans.gif%" AND post_type NOT IN ('revision')' # In PowerShell, you need to escape " wp db query 'SELECT id FROM wp_posts WHERE post_content LIKE \"%trans.gif%\" AND post_type NOT IN ('revision')'
3.10.17. search-replace wp-cli:search-replace
Search/replace intelligently handles PHP serialized data, and does not change primary key values.
# Search and replace but skip one column wp search-replace 'http://example.dev' 'http://example.com' --skip-columns=guid # dry run; only for specific tables wp search-replace 'foo' 'bar' wp_posts wp_postmeta wp_terms --dry-run # only for specific columns wp search-replace 'foo' 'bar' wp_posts --include-columns=post_title,post_content # Run case-insensitive regex search/replace operation (slow) wp search-replace '\[foo id="([0-9]+)"' '[bar id="\1"' --regex --regex-flags='i' # Turn your production multisite database into a local dev database wp search-replace --url=example.com example.com example.dev 'wp_*options' wp_blogs # Search/replace to a SQL file without transforming the database # Current database is not changed. The change result is output to a file wp search-replace foo bar --export=database.sql # Pantheon Terminus # For deletion, don't use '', use ' ' with space terminus remote:wp my-site.env -- search-replace 'abc' ' ' --dry-run # Don't put `;` in the search and replace strings. Use regex # Since regex is slow, add certain table to search and limit the columns terminus remote:wp my-site.env -- search-replace '(<script src="https:\/\/abc\.xyz\.net\/sub\/123_456_\/a\.js\?pid=.+" type="text\/javascript"><\/script>)' ' ' wp_posts --include-columns=post_content --skip-columns=guid --dry-run --regex # Bash script: Search/replace production to development url (multisite compatible) #!/bin/bash if $(wp --url=http://example.com core is-installed --network); then wp search-replace --url=http://example.com 'http://example.com' 'http://example.dev' --recurse-objects --network --skip-columns=guid --skip-tables=wp_users else wp search-replace 'http://example.com' 'http://example.dev' --recurse-objects --skip-columns=guid --skip-tables=wp_users fi
3.10.18. taxonomy
# default taxonomies are category and post_tag wp taxonomy list wp taxonomy get mytax_slug
3.10.19. term
wp term list category wp term list mytax_slug wp term delete category 15 wp term delete category apple --by=slug # delete all terms for a taxonomy post_tag wp term list post_tag --field=term_id | xargs wp term delete post_tag
3.10.20. eval
It doesn't work on Pantheon's Terminus
wp eval "echo rand(); echo 'abc';" --skip-wordpress --allow-root wp eval 'print(phpinfo());' --skip-wordpress --allow-root wp eval 'print(phpinfo());' --skip-wordpress --allow-root | grep max_input_vars
3.10.21. config list
3.10.22. embed wp-cli:embed
- Refer to wp:api:oembed
3.10.23. Common Issues
3.10.23.1. Can't connect to database
https://make.wordpress.org/cli/handbook/common-issues/#error-cant-connect-to-the-database
Make sure this line is exactly like this and wp-config.php shouldn't run any WP functions but only define constants
require_once(ABSPATH . 'wp-settings.php');
Latest version of WordPress Docker is PHP 7 and hence has some extensions removed like mysql_connect. Make sure your PHP environment is the same for WordPress and wp-cli.
3.10.23.2. Out of memory
php -i | grep php.ini # /usr/local/etc/php/conf.d/myphp.ini memory_limit=512M # then apache:restart # Or you can run this to temporarily change the memory limit php -d memory_limit=512M "$(which wp)" package install <package-name>
3.10.23.3. Allow root wp-cli:allow-root
This is for interactive shell
alias wp='wp --allow-root'
Create a bash script wrapper for already started processes
#!/bin/bash exec /usr/local/bin/wp2 "$@" --allow-root
3.11. URL, Current URL, and slug wp:current url
Current URL on your website php:parse_url
global $wp; echo $wp->request; // path e.g. /path/to/page echo home_url( $wp->request ); // works for pretty permalink echo add_query_arg( $wp->query_vars, home_url( $wp->request )); // regardless of permalink setting // Old method and may not work // $current_url = home_url(add_query_arg(array(), $wp->request)); // echo esc_url($current_url); $url = 'http://username:password@hostname:9090/path?arg=value#anchor'; $_url_array = parse_url($url); // keys are scheme, host, port, user, pass, path, query (after ?), fragment (after #) if ($_url_array !== false && !is_null($_url_array['host']) {} // Slug $_post = get_post(); if (!empty($_post) && $_post->post_name == 'your-slug') { // do something } // Site URL // http://google.com without trailing slash get_site_url(); // Custom Link echo '<a href="'.esc_url( 'somelink' ).'" target="_blank">'.__('some text', 'textdomain').'</a>';
get_permalink( int|WP_Post $post, bool $leavename = false )
- Depends on post type, it can call other functions
get_page_link()get_attachment_link()get_post_link()
pre_post_linkpost_link_categorypost_link
// Get term URL echo get_term_link( 'term-slug-or-term-obj-or-term-id', 'taxonomy-slug' );
3.12. WP_Post Post Object wp:post
$p = get_post(); echo $p->ID; // int
- https://developer.wordpress.org/reference/classes/wp_post/
- wp:db:wp_posts
- int
- string
- string
- pub date, string
2019-07-16 09:46:16 - pub date, string
2019-07-16 09:46:16wp:post:post_date_gmt- Convert to MySQL date, WP function
mysql2date( DATE_W3C, $post->post_date_gmt, false )is string2019-07-16T13:46:16+00:00
- Template methods are in category-template.php wp:t:category
3.12.1. has_term wp:f:has_term
// has_term( $term = '', $taxonomy = '', $post = null ) // $term :: name/term_id/slug or array of them
3.13. WP_User, user object wp:user
Public properties ID (int) - the user's ID. caps (array) - the individual capabilities the user has been given. cap_key (string) - roles (array) - the roles the user is part of. allcaps (array) - all capabilities the user has, including individual and role based. first_name (string) - first name of the user. last_name (string) - last name of the user.
Public Methods has_cap :: has role name or capability
if ( is_user_logged_in() ) { $user = wp_get_current_user(); // $user->ID public properties if ($user->has_cap('administrator')) { // user is admin } }
3.14. WP Custom Fields wp:Custom_Fields
3.14.1. Get Custom Fields
Normal
print_r(get_post_custom_keys()); // Get all keys as an array
print_r(get_post_custom()); // Get keys and values as an array
print_r(get_post_custom_values('fieldkey')); // Get values as an array
// get all meta
$meta = get_post_meta(get_the_ID());
global $post;
echo get_post_meta($post->ID, 'my-ad', true);
Plugin is different :: wp:plugin:types
3.14.2. Custom Fields in WP_Meta_Query
Custom fields created by wp:plugin:acf can use WP_Meta_Query as normal.
Custome fields creasted by wp:plugin:types, like a set of checkboxes, in WP_Query, you should use the LIKE operator. Refer to WP Types Custom Field Checkbox and WP Meta Query
3.14.3. Create Custom Fields for Custom Post Type
Use wp:plugin:acf
3.15. WP_Query The Loop wp:The_Loop wp:query:main
- The main query/loop is based on the URL request and is processed before templates are loaded
- The wp:conditional tags use the main query/loop
- The secondary query/loop is
new WP_Queryin theme template or plugin files - It's also called The Loop
- wp:template tags use the secondary query/loop
- By default, The Loop uses the main query's current post to set
global $post new WP_Query()andthe_post(called inhave_posts()) modify The Loop and henceglobal $postis modified- After these
new WP_Query()andthe_post(Loop functions) are used, run wp:f:wp_reset_postdata to reset global $post to the original new WP_Query()doesn't change wp:query:main- Refer to wp:global:wp_query for raw query
3.15.1. Syntax
<?php $query = new WP_Query('cat=-3,-8'); // Use query_posts($query_array | $string) instead of creating a new WP_Query then you don't have to prefix $query for methods // But query_posts change the global $wp_query ?> <?php if ( $query->have_posts() ) : while ( $query->have_posts() ) : $query->the_post(); ?> <?php // set the global $post to individual one in $query ?> <div><?php the_content(); ?></div> <?php if ( has_post_thumbnail() ): ?> <img src="<?php the_post_thumbnail_url('large'); ?>"> <?php the_post_thumbnail(); ?> <?php endif; ?> <?php endwhile; ?> <?php wp_reset_postdata(); // set back global $post to original ?> <?php endif; ?> <?php $query->rewind_posts(); // in order to loop $query again ?>
3.15.2. WP_Query
3.15.2.1. Basics
- Generator
- https://generatewp.com/wp_query/
- Class Reference
- https://codex.wordpress.org/Class_Reference/WP_Query
- (no term)
- Use objects in order
- Class methods and properties
wp-includes/class-wp-query.php- Debug
- wp:debug:wp_query
- Public vs Private query vars
- Refer to wp:filter:query_vars
$args = array( 'post_type' => array('editorial', 'blog'), // public string|array d:'post'. Refer to wp:f:register_post_type // 'any' can be used // If 'tax_query' is used, default is changed to 'any' 'post_status' => array('publish'), // string|array d:'publish'. Refer to wp:f:register_post_status // If user is logged in, 'private' is added. // Custom post statuses are added. // If is in admin, protected statuses are added: future, draft, pending // 'any' can be used 'posts_per_page' => -1, // int. -1 shows all posts. Doesn't work in feed (Default 10). Use wp:filter:post_limits 'post_mime_type' => 'image', // default:not set. string|array. e.g. `image/jpeg` | `image` (for all images) // $all_mimes = get_allowed_mime_types(); // $unsupported_mimes = ['image/jpeg', 'image/gif', 'image/png', 'image/bmp', 'image/tiff', 'image/x-icon' ]; // $accepted_mimes = array_diff( $all_mimes, $unsupported_mimes); // 'post_mime_type' => $accepted_mimes // Tag Parameters 'tag' => 'tag-slug', // public. Can be 'bread,baking' for one of or 'bread+backing' for all 'tag_id' => 123, // public 'tag__and' => [123,321], // private 'tag__in' => [123,321], // private 'tag__not_in' => [123,321], // private 'tag_slug__and' => ['tag1-slug','tag2-slug'], // private 'tag_slug__in' => ['tag1-slug','tag2-slug'], // private // Category Parameters 'cat' => '123', // public. Can be '2,6,17,38' for one of or '-12,-34,-56' for except 'category_name' => 'category-slug', // public. Can be 'staff,news' for one of or 'staff+news' for all 'category__and' => [123,321], // private 'category__in' => [123,321], // private 'category__not_in' => [123,321], // private 'meta_query' => $meta_query, // $meta_query is a nested array parsed by WP_Meta_Query // Taxonomy Parameters // 'tax' => '', deprecated 'tax_query' => $tax_query, // refer to WP Tax Query // Refer to WP Order and Orderby 'order' => 'DESC', // public d:'ASC' 'orderby' => 'date', // public d:'date' which is post_date. Others are publish_date, title 'suppress_filters' => false, // default. When true, filters will not run ); $query = new WP_Query($args); // or // $q = new WP_Query; // $q->query($args);
3.15.2.2. Get child posts
$args = [ 'post_type' => 'articles', 'post_parent' => get_the_ID(), 'posts_per_page' => -1, 'orderby' => 'menu_order' ]; $children = new WP_Query($args);
3.15.2.3. Get attached images
$images = get_attached_media( 'image' ); // get_attached_media( string $type, int|WP_Post $post = 0 ) // $type :: mime type // $post :: default is global $post // return array foreach ($images as $image) { ?> <img src="<?php echo wp_get_attachment_image_src($image->ID,'full'); ?>" /> <?php }
3.15.2.4. Loop the main loop
<?php if (have_posts()) : while (have_posts()) : the_post(); ?> <div <?php post_class(); ?> id="post-<?php the_ID(); ?>"> <h1><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h1> <?php the_content(); ?> </div> <?php endwhile; ?> <div class="navigation"> <div class="next-posts"><?php next_posts_link(); ?></div> <div class="prev-posts"><?php previous_posts_link(); ?></div> </div> <?php else : ?> <div <?php post_class(); ?> id="post-<?php the_ID(); ?>"> <h1>Not Found</h1> </div> <?php endif; ?>
3.15.2.5. Loop with query_posts() wp:f:query_posts
- Try to avoid using it. Use
new WP_Query()instead - It resets wp:global:wp_query
<?php global $query_string; // required $posts = query_posts($query_string.'&cat=-9'); // exclude category 9 // $posts = query_posts($query_string.'&posts_per_page=3&cat=-6,-9&order=ASC'); ?> <?php // DEFAULT LOOP GOES HERE ?> <?php wp_reset_query(); // reset the query ?>
3.15.2.6. Loop with new WP_Query()
<?php $custom_query = new WP_Query('cat=-9'); // exclude category 9 while($custom_query->have_posts()) : $custom_query->the_post(); // global $post is modified ?> <div <?php post_class(); ?> id="post-<?php the_ID(); ?>"> <h1><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h1> <?php the_content(); ?> </div> <?php endwhile; wp_reset_postdata(); // reset the query
Array join 2 query results
- Although php:array_merge is used, since 2 arrays have numeric keys, it's actually array join
<?php $args = array( 'post_type' => 'speaker', 'posts_per_page' => - 1, 'orderby' => 'meta_value', 'meta_key' => 'speaker_last_name', 'order' => 'ASC' ); $tax_query_keynote = array( array( 'taxonomy' => 'speaker_type', 'field' => 'slug', 'terms' => 'keynote-speakers', ) ); $tax_query_not_keynote = array( array( 'taxonomy' => 'speaker_type', 'field' => 'slug', 'terms' => 'keynote-speakers', 'operator' => 'NOT IN' ) ); $args_not_keynote = $args; $args['tax_query'] = $tax_query_keynote; $args_not_keynote['tax_query'] = $tax_query_not_keynote; $speakers_keynote = new WP_Query($args); $speakers_not_keynote = new WP_Query($args_not_keynote); $speakers = new WP_Query; $speakers->posts = array_merge($speakers_keynote->posts, $speakers_not_keynote->posts); $speakers->post_count = $speakers_keynote->post_count + $speakers_not_keynote->post_count; if ($speakers->have_posts()) { while ($speakers->have_posts()) { $speakers->the_post(); echo get_the_title(); the_title(); // the_*() functions can be used as global $post is set } }
3.15.2.7. Loop with get_posts()
- Use
new WP_Query(), the same$args - Return an array of WP_Post object wp:post
- most filters are not run
- It's good when
new WP_Query($args)does not work because of other plugins/theme have extra filters that mess up WP_Query - Very safe to use
global $post; // required $args = array('category' => -9); // exclude category 9 $custom_posts = get_posts($args); // have to use $post in foreach foreach($custom_posts as $post) { setup_postdata($post); // global $post is set and modified echo get_the_title(); } wp_reset_postdata(); // important // loop without modifying global $post foreach ($custom_posts as $p) { the_title( $speaker->ID ); } // don't need to reset postdata
3.15.2.8. Loop outside a WordPress Loop wp:loop:outside
if ( have_posts() ) { $items = []; while ( have_posts() ) { the_post(); $items[] = new Agenda( get_the_ID(), get_field( 'agenda_date' ), get_field( 'agenda_start_time' ), get_field( 'agenda_end_time' ), get_permalink() ); } // loop not using ->the_post foreach ($items as $item) { // some template functions you can use as it is echo get_the_title( $item->id ); // for excerpt, use this $excerpt = apply_filters('the_excerpt', get_post_field('post_excerpt', $item->id)); } }
3.15.2.9. Loop inside loop, Loop Stack wp:loop:stack
if ( have_posts() ) { // main loop for single speaker content type while ( have_posts() ) { the_post(); // global $post is set and modified; global $post; $bk_speaker_post = clone $post; // clone the single speaker post the_title(); // the main speaker title. `the_*()` always uses the global $post // ACF field contains agenda $sessions = get_field( 'speaker_agenda', false, false ); // loop the main speaker's agendas if ( $sessions ) { $sessions_sorted = new WP_Query( [ 'post_type' => 'agenda', 'posts_per_page' => - 1, 'posts__in' => $sessions ] ); // loop agenda's speakers if ( $sessions_sorted ) { while ( $sessions_sorted->have_posts() ) { $sessions_sorted->the_post(); // global $post is set and modified global $post; $bk_agenda_post = clone $post; the_title(); // agenda title echo get_the_title(); // agenda title echo get_the_title( $bk_speaker_post->ID ); // speaker title! // reverse query agenda to list speakers of this agenda $agenda_speakers = get_posts( [ 'post_type' => 'speaker', 'meta_query' => [ [ 'key' => 'speaker_agenda', 'value' => '"' . get_the_ID() . '"', // matches exactly "123", not just 123. This prevents a match for "1234" 'compare' => 'LIKE' ] ] ] ); foreach ( $agenda_speakers as $speaker ) { // global $post is not modified and it's the agenda $speakers_output[] = sprintf( '<a href="%s">%s</a>', get_permalink( $speaker->ID ), get_the_title( $speaker->ID ) ); } // since global $post is not changed, it is still the agenda post the_title(); // agenda title! not speaker title echo get_the_title(); // agenda title echo get_the_title( $bk_speaker_post->ID ); // the main speaker title! // let's create another loop but now we change the global $post // $post is global $post foreach ( $agenda_speakers as $post ) { setup_postdata( $post ); // $post is global $post $speakers_output[] = sprintf( '<a href="%s">%s</a>', get_permalink(), get_the_title() ); } the_title(); // the last speaker title of the current agenda! not agenda title as before echo get_the_title(); // the last speaker title of the current agenda! echo get_the_title( $bk_speaker_post->ID ); // the main speaker title! echo get_the_title( $bk_agenda_post->ID ); // the current agenda title } } // loop agenda's speakers. } // loop the main speaker's agendas. } // loop the main speaker }
3.15.2.10. is_*() methods wp:wp_query:m:is_*
is_home()is_front_page()- Site Front Page. I use this one
- (no term)
is_single( $post = '' )is_singular( $post_types = '' )- string|array
is_post_type_archive()is_post_type_archive('yourposttype')- If a custom tax archive page (taxonomy and/or a term) is being displayed
is_tax( $taxonomy= '' , $term = '')e.g.is_tax( 'my_tax' )is_tax( 'my_tax', 'my_term' )- (no term)
is_main_query()
3.15.2.11. found_posts
3.15.2.12. post_count
- Respect
posts_per_page
3.15.2.13. current_post
- Loop has not started or loop has just started
- -1
- (no term)
- First item in a loop is zero
- (no term)
Last post in a loop
global $wp_query; if ( ($wp_query->current_post + 1) == ($wp_query->post_count) ) { // last post }
3.15.2.14. in_the_loop Whether posts are in a loop
3.15.2.15. post Current post
3.15.2.16. posts List of posts
3.15.2.17. query Query vars set by the user
3.15.2.18. query_vars Query vars, after parsing wp:wp_query:query_vars
- Each query var is available in template files. Refer to wp:f:get_template_part
3.15.2.19. have_posts()
- At loop end
- wp:action:loop_end
$this->current_post = -1; $this->post = $this->posts[0];
- wp:action:loop_no_results
3.15.2.20. the_post()
$this->in_the_loop = true;$this->next_post()$this->setup_postdata( $post )
3.15.2.21. query( $query ) wp:wp_query:m:query
- $query
- array of post objects or post IDs
- (no term)
$q->init();- (no term)
$q->query = $this->query_vars = wp_parse_args( $query )- (no term)
- return wp:wp_query:m:get_posts
3.15.2.22. parse_query( $query = '' ) wp:wp_query:m:parse_query
- Sanitize wp:wp_query:query_vars
- Set
wp:wp_query:is_*properties - wp:action:parse_query
3.15.2.23. get_posts() wp:wp_query:m:get_posts
- Abstract
parse_query()- wp:wp_query:m:parse_query
- (no term)
- wp:action:pre_get_posts
- wp:filter:posts_search
- clause
WHEREfor Search Result SQL - wp:filter:posts_search_orderby
- clause
ORDER BYfor Search Result SQL - wp:filter:posts_where
- (no term)
- wp:filter:posts_join
- (no term)
- wp:filter:comment_feed_join
- (no term)
- wp:filter:comment_feed_where
- (no term)
- wp:filter:comment_feed_groupby
- (no term)
- wp:filter:comment_feed_orderby
- (no term)
- wp:filter:comment_feed_limits
- (no term)
- wp:filter:posts_where_paged
- (no term)
- wp:filter:posts_groupby
- (no term)
- wp:filter:posts_join_paged
- (no term)
- wp:filter:posts_orderby
- (no term)
- wp:filter:posts_distinct
- (no term)
- wp:filter:post_limits
- (no term)
- wp:filter:posts_fields
- (no term)
- wp:filter:posts_clauses
- wp:action:posts_selection
- for caching plugins
- wp:filter:posts_where_request
- for caching plugins
- wp:filter:posts_groupby_request
- for caching plugins
- wp:filter:posts_join_request
- for caching plugins
- wp:filter:posts_orderby_request
- for caching plugins
- wp:filter:posts_distinct_request
- for caching plugins
- wp:filter:posts_fields_request
- for caching plugins
- wp:filter:post_limits_request
- for caching plugins
- wp:filter:posts_clauses_request
- for caching plugins
- wp:filter:posts_request
- completed SQL query
- (no term)
- wp:filter:posts_pre_query
- (no term)
- wp:filter:split_the_query
- wp:filter:posts_request_ids
- (no term)
- wp:filter:posts_results
- (no term)
- wp:filter:comment_feed_join
- (no term)
- wp:filter:comment_feed_where
- (no term)
- wp:filter:comment_feed_groupby
- (no term)
- wp:filter:comment_feed_orderby
- (no term)
- wp:filter:comment_feed_limits
- (no term)
- wp:filter:the_preview
- (no term)
- wp:filter:the_posts
- (no term)
- Other filters called in other WP_Query functions
- pre_option_posts_per_rss
- wp:filter:pre_option_*
- (no term)
- wp:filter:found_posts
3.15.3. WP_Meta_Query
https://codex.wordpress.org/Class_Reference/WP_Meta_Query https://rudrastyh.com/wordpress/meta_query.html
- Refer to wp-cli:post
- wp:filter:get_meta_sql
If only one condition is needed, use these fields
- meta_key
- string
- meta_value
- string|array
- meta_type
- string
- meta_compare
- string
$rd_args = array( 'meta_key' => 'show_on_homepage', 'meta_value' => 'on', 'meta_compare' => '!=' ); $rd_query = new WP_Query( $rd_args );
Or use meta_query for nesting multiple conditions
3.15.3.1. meta_query Nesting
Nested: key1=value1 OR (key2=value2 AND key3=value3)
$meta_args = array( 'relation' => 'OR', // Optional. Default is 'AND' array( 'key' => 'key1', 'value' => 'value1', 'compare' => '=', //default ), array( 'relation' => 'AND', array( 'key' => 'key2', 'value' => 'value2', 'compare' => '=', ), array( 'key' => 'key3', 'value' => 'value3', 'compare' => '=', ), ), ); $meta_query = new WP_Meta_Query($meta_args); // parse args array
To get posts that have featured images
$args = array( 'post_type' => array( 'news', 'videos' ), 'meta_query' => array( array('key'=> '_thumbnail_id') ), 'posts_per_page' => 4, ); $gamechangers = new WP_Query( $args );
3.15.3.2. Named sub-meta queries and multiple orderby arguments
- The following requires all sort columns have values
- The best to solve sort column might not have values is to make sure all posts have values for that sort column
- Refer to wp-cli:post meta
- Refer to wp:filter:get_meta_sql to sort columns which might not have values or exist
$q = new WP_Query( array( 'meta_query' => array( 'relation' => 'AND', 'state_clause' => array( 'key' => 'state', 'value' => 'Wisconsin', ), 'city_clause' => array( 'key' => 'city', 'compare' => 'EXISTS', ), ), 'orderby' => array( 'city_clause' => 'ASC', // 'key' is used to order 'state_clause' => 'DESC', ), ) );
3.15.3.3. Review full query
$q_args = array(...); $meta_query = new WP_Meta_Query(); $meta_query->parse_query_vars( $q_args ); $mq_sql = $meta_query->get_sql( 'post', // post, comment, user $wpdb->posts, // the table to look for rows 'ID', // for posts, it's ID, for comments it's comment_ID, for users is ID null // $context (obj) optional ); // return array (size=2) 'join' => string ' INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)' (length=62) 'where' => string ' AND (wp_postmeta.meta_key = 'some_key' )' (length=40)
3.15.3.4. Operators: compare and type
compare
=,!=>,>=,<,<=- LIKE, NOT LIKE
- IN, NOT IN
- BETWEEN, NOT BETWEEN
- EXISTS, NOT EXISTS
- mysql:regex
RLIKE(synonym of REGEXP)
type In MySQL, it means CAST(the field, to a type)
- NUMERIC
- BINARY
- CHAR
- works with 'compare' value BETWEEN only if the date is stored as YYYY-MM-DD
- DATETIME
- DECIMAL
- MySQL Integer
- mysql:datatype:datetime
WP Types Custom Field Checkbox and WP Meta Query
Say the wpcf-newsletters custom field created by WP Types has 2 checkboxes are selected: enligne and pw.
The following 2 strings will appear in the serialized json string as value in WP_Meta_Query
a:1:{i:0;s:7:"enligne";}
a:1:{i:0;s:2:"pw";}
Select the posts that have checkbox enligne checked.
$meta_query = array( array( 'key' => 'wpcf-newsletters', 'value' => 'a:1:{i:0;s:7:"enligne";}', 'compare' => 'LIKE', ), );
3.15.4. WP Tax Query
- https://codex.wordpress.org/Class_Reference/WP_Query#Taxonomy_Parameters
- https://generatewp.com/wp_tax_query/
$tax_query = array( 'relation' => 'AND', // Optional array(), array( 'taxonomy' => '', 'field' => 'term_id', // string :: term_id (default), name, slug or term_taxonomy_id 'terms' => 'bob', // int/string/array :: multiple terms 'operator' => 'IN', // string :: IN (default), NOT IN, AND, EXISTS, NOT EXISTS ), );
3.15.5. WP Order and Orderby
3.15.5.1. order
- string or array to match orderby
- default
- ASC
3.15.5.2. orderby
- Refer to WP_Meta_Query for multiple orderby
- default, which is post_date
- modified date
- post id
- parent id
- random order
- comment_count
- author
- title
- post name post slug
- post type
- https://developer.wordpress.org/reference/classes/wp_query/#order-orderby-parameters
3.15.6. WP_Term_Query wp:WP_Term_Query
// get parent term by slug $sponsor_type = get_term_by( 'slug', 'sponsors', 'sponsor_type' ); if ( ! $sponsor_type ) { return; } // get child terms and sort by ACF field sponsorship_type_order $args = [ 'taxonomy' => 'sponsor_type', 'parent' => $sponsor_type->term_id, 'order' => 'sponsorship_type_order', 'orderby' => 'ASC', 'hide_empty' => true, // default true. false to return terms that have no post associated with it ]; $sponsor_types = new WP_Term_Query( $args ); foreach ( $sponsor_types->terms as $type_term ) { // get posts that has this term $args = [ 'post_type' => 'sponsors', 'post_status' => 'publish', 'posts_per_page' => - 1, 'tax_query' => [ [ 'taxonomy' => 'sponsor_type', 'field' => 'slug', 'terms' => $type_term->name ] ], 'orderby' => 'title', ]; $sponsors_by_type = new WP_Query( $args ); if ( $sponsors_by_type->have_posts() ) : ?> <h2 class="mb-5"><?php echo $type_term->name; ?></h2> <?php while ( $sponsors_by_type->have_posts() ) : $sponsors_by_type->the_post(); ?> <article class="row mb-3"> <div class="col-md-3 align-self-center"> <a href="<?php the_permalink() ?>"> <?php if ( has_post_thumbnail() ) { the_post_thumbnail( 'thumbnail', [ 'class' => 'd-block img-fluid mx-auto mb-3', 'alt' => esc_attr( get_the_title() ), ] ); } else { echo '<p class="h3 text-center">' . get_the_title() . '</p>'; } ?> </a> </div> <div class="col-md-9"> <?php the_content(); ?> </div> </article> <?php endwhile; wp_reset_postdata(); ?> <?php endif; }
3.15.7. Review Raw Query
$args=[]; $q = new WP_Query($args); echo "<pre>"; print_r($q->request); echo "</pre>";
SELECT wp_posts.* FROM wp_posts -- tax_query uses LEFT JOIN LEFT JOIN wp_term_relationships ON (wp_posts.ID = wp_term_relationships.object_id) -- meta_query by default uses INNER JOIN INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) INNER JOIN wp_postmeta AS mt1 ON ( wp_posts.ID = mt1.post_id ) -- meta_query when there's `NOT EXISTS` then all INNER JOIN are changed to LEFT JOIN --LEFT JOIN wp_postmeta AS mt1 ON ( wp_posts.ID = mt1.post_id AND mt1.meta_key= 'custom_rder') WHERE 1=1 AND ( wp_term_relationships.term_taxonomy_id IN (16896) ) AND ( wp_postmeta.meta_key = 'speaker_last_name' AND mt1.meta_key = 'custom_order' ) AND wp_posts.post_type = 'speaker' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled') GROUP BY wp_posts.ID ORDER BY CAST(mt1.meta_value AS CHAR) ASC, CAST(wp_postmeta.meta_value AS CHAR) ASC
3.16. Functions
3.16.1. Plugin API wp-includes/plugin.php wp:api:plugin
3.16.1.1. add_filter, apply_filters, apply_filters_ref_array wp:add_filter
Syntax, remove_filter
- Modify internal data / function returned
- For example, WP_Query gets internal settings for building a query
- You can temporary add_filter to change the internal settings, then build a WP_Query, later remove_filter
- so that you only change the internal setting for that query building without affecting other usage
add_filter( 'tagname', 'lili_funct', 10, // int. Priority, lower number sooner 2, // int. Accepted args, default is 1! Be careful ); apply_filters( 'tagname', $result, $var1); // callback should take 2 variables function lili_funct($result, $var1) { $new = $result . 'abc'; return $new; } remove_filter( 'tagname', 'lili_funct', 10 // priority, must match the add_filter priority, default is 10 );
function remove_filter_the_title() { remove_filter('the_title', 'before_post_title', 10); } add_action( 'wp_head', 'remove_filter_the_title', 1 ); function add_filter_the_title() { add_filter('the_title', 'before_post_title', 10, 2); } add_action( 'wp_head', 'add_filter_the_title', 3 );
Class method
// Static class method add_filter( 'media_upload_newtab', array( 'My_Class', 'media_upload_callback' ) ); // Instance method add_filter( 'media_upload_newtab', array( $this, 'media_upload_callback' ) ); // Anonymous function add_filter( 'the_title', function( $title ) { return '<strong>' . $title . '</strong>'; } );
3.16.1.2. add_action, do_action, do_action_ref_array wp:add_action
Event based
add_action( string $tag, string $function_name, int $priority = 10, int $accepted_args = 1 // number of args the function accepts ); // add_action( 'atag' is called by do_action( 'atag' or do_action_ref_array( 'atag' // child theme add_action( 'abc', 'xyz' ); // parent theme add_action( 'abc', 'abc' ); // xyz and abc will run in order do_action( 'abc' ); // child theme remove_action( 'abc', 'ijk' ); // parent theme add_action( 'abc', 'ijk'); // ijk is added and then removed, so ijk will not run do_action( 'abc' );
- Action sequence wp:action:loading sequence
- https://codex.wordpress.org/Plugin_API/Action_Reference
- http://lance.bio/2017/10/11/wordpress-hooks-and-filters-order-of-precedence/
- (ref array) after WP object is set up wp:action:wp
WP_Customize_Widgets->customize_register
3.16.1.3. register_activation_hook wp:f:register_activation_hook
register_activation_hook( string $file, callable $function)- wp:add_action with name
$fileand add the callback function
- wp:add_action with name
register_deactivation_hook
// run something after a plugin is activated // usually the code is added in wp-content/plugins/myplugin/myplugin.php, but it can be anywhere /** wp-content/plugins/myplugin/myplugin.php * Register Task post type. */ require_once plugin_dir_path( __FILE__ ) . 'includes/posttypes.php'; register_activation_hook( __FILE__, 'taskbook_rewrite_flush' ); /** wp-content/plugins/myplugin/includes/posttypes.php * Register Task post type. */ function taskbook_cpt_init() { register_post_type( 'task', $args ); } add_action( 'init', 'taskbook_cpt_init' ); function taskbook_rewrite_flush() { // First, we "add" the custom post type via the above written function. // Note: "add" is written with quotes, as CPTs don't get added to the DB, // They are only referenced in the post_type column with a post entry, // when you add a post of this CPT. taskbook_cpt_init(); // ATTENTION: This is *only* done during plugin activation hook in this example! // You should *NEVER EVER* do this on every page load!! flush_rewrite_rules(); }
3.16.2. Transients API - set_transient vs wp_cache_set wp:cache
WordPress cache API in database.
DELETE FROM wp_options WHERE option_name LIKE ('%\_transient\_%')
WP object cache stores objects and primitives but not in a persistent manner by default. This means caching happens in memory and it only lives for the request's lifetime cycle (e.g. one pageload). Redis plugin can make it persistent.
https://codex.wordpress.org/Class_Reference/WP_Object_Cache
Transient API saves variables, arrays, objects tied with an expiration date on db and has persistent object caching. However when cached objects expire, they remain on db which needs to be pruned every once in a while.
WP detects if persistent object cache, if so, bypass Transients API and route to WP object cache.
// wp_cache_get( int|string $key, string $group = '', bool $force = false, bool $found = null ) // wp_cache_set( int|string $key, mixed $data, string $group = '', int $expire ) $result = wp_cache_get( 'my_result', 'my-group' ); if ( false === $result ) { $result = $wpdb->get_results( $query ); wp_cache_set( 'my_result', $result, 'my-group', 600 ); // expire after 5 mins } // Do something with $result;
3.16.3. Formatting wp-includes/formatting.php
3.16.3.1. wp_strip_all_tags( $string, $remove_breaks = false )
function wp_strip_all_tags($string, $remove_breaks = false) { $string = preg_replace( '@<(script|style)[^>]*?>.*?</\\1>@si', '', $string ); $string = strip_tags($string); if ( $remove_breaks ) $string = preg_replace('/[\r\n\t ]+/', ' ', $string); return trim( $string ); }
3.16.3.2. wp_trim_words( $text, $num_words = 55, $more = null ) wp:ellipsis
- Strip HTML and PHP tags and truncate string in words
function wp_trim_words( $text, $num_words = 55, $more = null ) { if ( null === $more ) { $more = __( '…' ); } $original_text = $text; $text = wp_strip_all_tags( $text ); /* * translators: If your word count is based on single characters (e.g. East Asian characters), * enter 'characters_excluding_spaces' or 'characters_including_spaces'. Otherwise, enter 'words'. * Do not translate into your own language. */ if ( strpos( _x( 'words', 'Word count type. Do not translate!' ), 'characters' ) === 0 && preg_match( '/^utf\-?8$/i', get_option( 'blog_charset' ) ) ) { $text = trim( preg_replace( "/[\n\r\t ]+/", ' ', $text ), ' ' ); preg_match_all( '/./u', $text, $words_array ); $words_array = array_slice( $words_array[0], 0, $num_words + 1 ); $sep = ''; } else { $words_array = preg_split( "/[\n\r\t ]+/", $text, $num_words + 1, PREG_SPLIT_NO_EMPTY ); $sep = ' '; } if ( count( $words_array ) > $num_words ) { array_pop( $words_array ); $text = implode( $sep, $words_array ); $text = $text . $more; } else { $text = implode( $sep, $words_array ); } /** * Filters the text content after words have been trimmed. * * @since 3.3.0 * * @param string $text The trimmed text. * @param int $num_words The number of words to trim the text to. Default 55. * @param string $more An optional string to append to the end of the trimmed text, e.g. …. * @param string $original_text The text before it was trimmed. */ return apply_filters( 'wp_trim_words', $text, $num_words, $more, $original_text ); }
3.16.3.3. antispambot( $email_address, $hex_encoding = 0 ) : string wp:f:antispambot
Encode email address to hex
function hide_email_addresses_sc($atts, $content = null) { // $atts['var1'], the following just sets a default if var1 attribute is not defined in shortcode $emailaddress_fields = shortcode_atts(array( 'id' => '', 'email' => '' ),$atts); $userid = $emailaddress_fields['id']; $e_mail = $emailaddress_fields['email']; if ($userid !=='' && $e_mail =='') { $emailaddress = get_the_author_meta('user_email',$userid); return '<a href="mailto:'.antispambot($emailaddress).'">'.antispambot($emailaddress).'</a>'; } if ($userid =='' && $e_mail !=='') { return '<a href="mailto:'.antispambot($e_mail).'">'.antispambot($e_mail).'</a>'; } else { return ''; } } add_shortcode('hide-email','hide_email_addresses_sc');
3.16.3.4. wpautop( $pee, $br = true ) : string wp:f:wpautop
- Used in
- wp:filter:the_content
- wp:filter:the_excerpt
- Change double line-breaks in text into
<p>...</p>
Turn off wpautop for specific pages
function remove_p_on_pages() { if ( is_page() ) { remove_filter( 'the_content', 'wpautop' ); remove_filter( 'the_excerpt', 'wpautop' ); } } add_action( 'wp_head', 'remove_p_on_pages' );
3.16.4. Main API wp-includes/functions.php wp:api:main
3.16.4.1. Option API wp:api:option
3.16.4.2. wp_parse_args( $args, $defaults = '' ) wp:f:wp_parse_args
Merge user defined arguments into defaults array
3.16.4.3. __return_*()
__return_true()__return_false()__return_empty_array()__return_null()__return_empty_string()
3.16.4.4. wp_upload_dir
Return an array to describe location of wp upload directory
array ( 'path' => '/var/www/html/wp-content/uploads/2018/09', // next upload will go to this path 'url' => 'http://localhost:32006/wp-content/uploads/2018/09', // next upload 'subdir' => '/2018/09', 'basedir' => '/var/www/html/wp-content/uploads', 'baseurl' => 'http://localhost:32006/wp-content/uploads', 'error' => false )
3.16.4.5. do_feed() wp:f:do_feed
do_feed()- Get feedname from
get_query_var('feed')and do wp:action:do_feed_$feedname - Default added feed actions
do_feed_rdf()wp-includes/feed-rdf.phpdo_feed_rss()wp-includes/feed-rss.phpdo_feed_rss2( $for_comments )wp-includes/feed-rss2-comments.phporfeed-rss2.phpdo_feed_atom( $for_comments )wp-includes/feed-atom-comments.phporfeed-atom.php
- Get feedname from
3.16.5. Translate wp:translate
Use tool:Poedit
3.16.5.1. Translate Functions wp-includes/l10n.php
- $text
- can't be a variable. And use single quotes instead of
"" - (no term)
- Use
sprintf(return) orprintf()(echo) for subsitutions. The msgid is"You have chosen the %s theme"
- Sample
echo sprintf( __( 'You have chosen the %s theme.' ) , the_color() ); printf( __( 'You have chosen the %s theme.' ) , the_color() ); echo __('some text', 'textdomain'); // return // 'textdomain' is optional but it has to be a string not a variable! Default it's 'default' // _e() is the same as __() but echoes // Plural or single // _n( $single, $plural, $number, $domain = 'default' ) echo _n('There is a comment', 'There are comments', get_comments_number()); _nx( $single, $plural, $number, $context, $domain = 'default' ); // Context echo _x('post a link', 'A link to the post', 'my-text-domain'); // 'A link to the post' matches msgctxt "A link to the post" in .po file. // _ex is the same as _x but echoes // return value for an HTML attribute printf( esc_attr__( 'View all posts filed under %1$s', 'my-text-domain' ), get_cat_name( 32 ) ); // echo value for an HTML attribute esc_attr_e( $text, $domain = 'default' ); esc_attr_x( $text, $context, $domain = 'default' ); // return value for HTML output esc_html__( $text, $domain = 'default' ); esc_html_e( $text, $domain = 'default' ); esc_html_x( $text, $context, $domain = 'default' );
3.16.5.2. Setup, .po and .mo files
Load textdomain in child theme which overwrites the textdomain defined in parent theme Do this in theme functions.php
function smart_mag_child_theme_locale() { $_r = load_child_theme_textdomain( 'textdomain-name', get_stylesheet_directory() . '/languages' ); // textdomain-name can be defined in parent theme or a new textdomain // if textdomain-name has been defined earlier, then this will overwrite // Check if textdomain inclusion is successful //if ( $_r ) var_dump( 'success!' ); if ( is_admin() ) { // load another textdomain } } add_action( 'after_setup_theme', 'smart_mag_child_theme_locale' );
Usually translate-ready theme will provide a .pot file which is a template file for .po and .mo files. Open .pot in tool:Poedit, generate a new translation. What Poedit saves is 2 files: .po and .mo.
Open .po file later, make changes and save to overwrite .po and .mo files. File .pot is no longer needed.
Catalog > Properties > Translation properties Specifiy the language Plural Forms by default is
nplurals=2; plural=n != 1;
Might need to specify the path in code for .po file. '.' means the current path.
"X-Poedit-SearchPath-0: .\n"
Catalog > Properties > Sources keywords contain all the Wordpress translation functions
__ _e _n:1,2 _x:1,2c _ex:1,2c
:1,2 indicates the function has 2 parts. 2c means the 2nd argument is a context/comment.
tool:Poedit can help you change most of the fields except that it can't specify context msgctxt.
For those situations, directly edit .po and open .po in Poedit and save to get the .mo file.
Some examples
Comments :: #. Author of the plugin/theme Due to emacs, ;; #. is #. in the following examples
Reference :: #: woocommerce/loop/orderby.php:23 Due to emacs, ;; #: is #: in the following examples
Concatenate multiple lines
#: 404.php:18 msgid "" "We're sorry, but we can't find the page you were looking for. It's probably " "some thing we've done wrong but now we know about it and we'll try to fix " "it. In the meantime, try one of these options:" msgstr ""
sprintf and printf subsitutions
#: archive.php:101 msgid "Yearly Archives: %s" msgstr ""
Context
#: bbpress/auth-modal.php:69 msgctxt "bbPress" msgid "Favorites" msgstr ""
Complex context
#: comments.php:53 msgid "" "Logged in as <a href=\"%1$s\">%2$s</a>. <a href=\"%3$s\" title=\"Log out of " "this account\">Log out?</a>" msgstr ""
Poedit save fr_CA.po and fr_CA.mo files. Copy and place them into the right place and then on Wordpress Settings > General > Site Lanuage to match fr_CA
3.16.6. User Role & Capabilities API wp:api:role-capabilities wp:roles
- https://codex.wordpress.org/Roles_and_Capabilities
Capabilities
$role = get_role('editor'); // WP_Role $role->add_cap('edit_theme_options'); // later if ( current_user_can( 'edit_theme_options' ) ) {}
- manage_options
- default admin only. Many plugins use this to determine if the user can make plugin specific changes
- edit_theme_options
- Add Admin Panel options for Appearance:
- Widgets
- Menus
- Customize
- Background
- Header
- (no term)
Post type capabilities
'capabilities' => array( 'edit_post' => 'edit_book', 'read_post' => 'read_book', 'delete_post' => 'delete_book', 'edit_posts' => 'edit_books', 'edit_others_posts' => 'edit_others_books', 'publish_posts' => 'publish_books', 'read_private_posts' => 'read_private_books', 'create_posts' => 'edit_books', ) [cap] => stdClass Object ( // Meta capabilities [edit_post] => "edit_{$capability_type}" [read_post] => "read_{$capability_type}" [delete_post] => "delete_{$capability_type}" // Primitive capabilities used outside of map_meta_cap(): [edit_posts] => "edit_{$capability_type}s" [edit_others_posts] => "edit_others_{$capability_type}s" [publish_posts] => "publish_{$capability_type}s" [read_private_posts] => "read_private_{$capability_type}s" // Primitive capabilities used within map_meta_cap(): [read] => "read", [delete_posts] => "delete_{$capability_type}s" [delete_private_posts] => "delete_private_{$capability_type}s" [delete_published_posts] => "delete_published_{$capability_type}s" [delete_others_posts] => "delete_others_{$capability_type}s" [edit_private_posts] => "edit_private_{$capability_type}s" [edit_published_posts] => "edit_published_{$capability_type}s" [create_posts] => "edit_{$capability_type}s" )
3.16.6.1. add_role( string $role, string $display_name, $capabilities = array() ) : WP_Role|null wp:f:add_role
$result = add_role( 'basic_contributor', __( 'Basic Contributor' ), array( 'read' => true, // true allows this capability 'edit_posts' => true, 'delete_posts' => false, // Use false to explicitly deny ) ); if ( null !== $result ) { echo 'Yay! New role created!'; } else { echo 'Oh... the basic_contributor role already exists.'; } // Create a new role when a plugin is activated function add_roles_on_plugin_activation() { add_role( 'custom_role', 'Custom Subscriber', array( 'read' => true, 'edit_posts' => true ) ); } register_activation_hook( __FILE__, 'add_roles_on_plugin_activation' );
3.16.7. Query API wp:api:query
- wrapper of
glboal $wp_querye.g.is_*functions query_posts( $query )wp:f:query_posts
3.16.7.1. Conditional Tags wp:conditional tags
- These call
global $wp_querywp:wp_query:m:is_*- Usually the wp:query:main
is_page( int|string|array $page = '' )
is_page(); // any page is_page(42); is_page( 'mypageslug' ); is_page( array(42, 'about-me', 'Page title' ) ); // either one of those id, slug or title is_page( array(42, 54, 6) );
is_page_template( string|array $template = '' )
is_page_template(); // any page template is_page_template( 'about.php' ); // another template 'page-templates/about.php'
is_single( int|string|array $post = '' )
- works for any post type except attachments and pages
- Post ID, title, slug or array of such (one of the specified)
is_singular( string|array $post_types = '' )
- $post_types
- Whether the query is for an existing single post of any post type. Or if $post_types is specified, additionally check if the query is for one of the Post Types specified
is_singular(); is_singular( 'article' ); is_singular( array( 'newspaper', 'book' ) );
is_admin_bar_showing()
3.16.7.2. wp_reset_postdata wp:f:wp_reset_postdata
- Use it when custom query
new WP_Query()is defined and used - After
global $postis modified by wp:query:secondary, reset it to the original that was set by wp:query:main
<?php // example args $args = array( 'posts_per_page' => 3 ); // the query $the_query = new WP_Query( $args ); ?> <?php if ( $the_query->have_posts() ) : ?> <!-- start of the loop --> <?php while ( $the_query->have_posts() ) : $the_query->the_post(); ?> <?php the_title(); ?> <?php the_excerpt(); ?> <?php endwhile; ?><!-- end of the loop --> <!-- put pagination functions here --> <?php wp_reset_postdata(); ?> <?php else: ?> <p><?php _e( 'Sorry, no posts matched your criteria.' ); ?></p> <?php endif; ?>
3.16.7.3. wp_reset_query wp:f:wp_reset_query
- Should be called after
query_posts() $GLOBALS['wp_query'] = $GLOBALS['wp_the_query'];andwp_reset_postdata()
Avoid to use it. To alter main query parameters before it's made, use wp:action:pre_get_posts
$args = array ( 'post_parent' => 5 ); query_posts( $args ); if ( have_posts() ): while ( have_posts() ) : the_post(); // Do stuff with the post content. the_title(); the_permalink(); // Etc. endwhile; else: // Insert any content or load a template for no posts found. endif; wp_reset_query();
3.16.8. Theme functions wp-includes/theme.php
3.16.8.1. get_template_directory, get_template_directory_uri, get_stylesheet_directory_uri, get_stylesheet_uri
get_template_directory()- the filesystem path of the current parent theme
get_template_directory_uri- URI e.g. protocol://domain of the current parent theme
get_stylesheet_directory()- the filesystem path of the current child theme
get_stylesheet_directory_uri- URI of the stylesheet directory. grab child and then parent theme.
get_stylesheet_uri- URI of style.css
include( get_template_directory() . '/includes/myfile.php'); wp_enqueue_script( 'twentyfifteen-skip-link-focus-fix', get_template_directory_uri() . '/js/skip-link-focus-fix.js', array(), '20141010', true );
3.16.8.2. get_theme_mod wp:f:get_theme_mod
get_theme_mod( $name, $default = false )
- Calls wp:f:get_theme_mods and gets wp:options:theme_modstheme_slug
- Returns wp:filter:theme_modtheme_slug
3.16.8.3. add_theme_support, remove_theme_support, current_theme_supports wp:f:add_theme_support
wp:f:current_theme_supports wp:f:add_theme_support:post-thumbnails
- Either in theme's
functions.phpor added in wp:action:after_setup_theme because it has to be loaded early - All theme support features
- Some features e.g.
html5usearray_merge - Other and custom features just take the latest call values as the feature
add_theme_support( 'post-thumbnails' ); // refer to wp:f:add_image_size add_theme_support( 'automatic-feed-links' ); // adds //<link rel="alternate" type="application/rss+xml" title="Website Name Feed" href="http://myweb.io/feed/" /> //<link rel="alternate" type="application/rss+xml" title="Website Name Comments Feed" href="http://myweb.io/comments/feed/" /> // If it's a page, add one more // <link rel="alternate" type="application/rss+xml" title="Website Name Page Name Comments Feed" href="http://myweb.io/2016/07/20/page-name/feed/" /> add_theme_support( 'title-tag' ); // wp will provide <title> not hard coded in template file using wp_title() add_theme_support( 'html5', array( 'search-form', 'comment-form', 'comment-list', 'gallery', 'caption', ) ); // use the default html5 markup instead of the default html4 file add_theme_support( 'post-formats', array( 'aside', 'image', 'video', 'quote', 'link', 'gallery', 'audio', ) ); // wp provides a list of formats for admin users to choose to display a post // refer to wp:f:has_post_format add_theme_support( 'custom-logo', array( 'width' => 250, 'height' => 250, 'flex-width' => true, ) ); // provide a place to upload theme logo in wp-admin // https://developer.wordpress.org/themes/functionality/custom-logo/ // refer to wp:custom logo:display add_theme_support( 'customize-selective-refresh-widgets' ); // https://make.wordpress.org/core/2016/03/22/implementing-selective-refresh-support-for-widgets/ add_theme_support( 'starter-content', $starter_content ); // https://make.wordpress.org/core/2016/11/30/starter-content-for-themes-in-4-7/ //add_theme_support( 'custom-background' ); // simple enable $defaults = array( 'default-color' => '', 'default-image' => '', 'default-repeat' => 'repeat', 'default-position-x' => 'left', 'default-position-y' => 'top', 'default-size' => 'auto', 'default-attachment' => 'scroll', 'wp-head-callback' => '_custom_background_cb', 'admin-head-callback' => '', 'admin-preview-callback' => '' ); add_theme_support( 'custom-background', $defaults ); // https://codex.wordpress.org/Custom_Backgrounds
3.16.8.4. add_editor_style( array|string $stylesheet = 'editor-style.css' )
- Used in
- Add stylesheets for TinyMCE
- Stylesheet name or array thereof, relative to theme root
- This function automatically adds another stylesheet with -rtl prefix, e.g. editor-style-rtl.css. If that file doesn’t exist, it is removed before adding the stylesheet(s) to TinyMCE. If an array of stylesheets is passed to add_editor_style(), RTL is only added for the first stylesheet.
- Since version 3.4 the TinyMCE body has .rtl CSS class. It's a better option to use that class and add any RTL styles to the main stylesheet
add_editor_style( array( 'assets/css/editor-style.css', twentyseventeen_fonts_url() ) );
3.16.8.5. get_theme_root, get_theme_root_uri
$theme_root = get_theme_root(); $files_array = glob("$theme_root/*", GLOB_ONLYDIR); echo "There are " . count($files_array) . " subdirectories in the " . $theme_root . " directory"; // There are 5 subdirectories in the /home/user/public_html/wp-content/themes directory. echo get_theme_root_uri(); // http://www.wordpress.org/wp-content/themes
3.16.9. Template Tags wp:template tags
wp-includes/xxx-template.phpwp-includes/feed.php- https://codex.wordpress.org/Template_Tags
3.16.9.1. post-thumbnail-template.php wp:t:thumbnail
has_post_thumbnail( $post = null)- return bool
get_post_thumbnail_id( $post = null, $size = 'post-thumbnail', $attr = '' );- get thumbnail id
get_the_post_thumbnail_*()- can specify which post
wp_get_attachment_image_srcis called- return html with img attributes
- srcset
wp_calculate_image_srcset- sizes
wp_calculate_image_sizes
- string|false
the_post_thumbnail_*- echo instead of return. calls
get_the_post_thumbnail_*the_post_thumbnail( $size, $attr = '' )echo get_the_post_thumbnail( null, $size, $attr )- e.g.
the_post_thumbnail( 'medium-large', array( 'class' => 'my-class') )
- e.g.
the_post_thumbnail_url( $size )- echo url
the_post_thumbnail_caption( $post = null )- echo thumbnail caption
wp_get_attachment_image_*()wp_get_attachment_image_srcis calledwp_get_attachment_image_src( $attachment_id, string|array $size = 'thumbnail', $icon = false )- just like
the_post_thumbnailbut returns an array [url, width, height, is_intermediate]. url is$thumb[0] wp_get_attachment_image( $attachment_id, string|array $size = 'thumbnail', $icon = false, string|array $attr = '' )- return HTML img element or empty string on failure
wp_get_attachment_image_url( $attachment_id, $size = 'thumbnail', $icon = false )- return string|false Attachment URL or false if no image is available
wp_get_attachment_image_srcset( $attachment_id, $size = 'medium', $image_meta = null )srcsetattrbiute value string or falsewp_get_attachment_image_sizes( $attachment_id, $size = 'medium', $image_meta = null )sizesattribute value string or false
wp_get_attachment_*()wp_get_attachment_caption( $post_id = 0 )$post_iddefault is global $post. Usually it's the attachment IDwp_get_attachment_metadata( $attachment_id = 0, $unfiltered = false )- array or false
wp_get_attachment_url( $attachment_id = 0)wp_get_attachment_thumb_file( $post_id = 0 )wp_get_attachment_thumb_url( $post_id = 0)
- Get alt
$alt = get_post_meta( get_post_thumbnail_id(), '_wp_attachment_image_alt', true);- (no term)
- Refer to wp:f:add_image_size
- NOTICE
- If the thumbnail has no physical file that has corresponding size, the_post_thumbnail_url will get the original image file, the_post_thumbnail will use the original file with in src and apply width/height attributes in HTML to achieve the purpose.
- 'post-thumbnail'
- default
- array(100, 100)
- Grab the biggest size in width and height image file possible.
- array(100, 9999)
- Grab the biggest size in width and height image file possible.
- (no term)
- Resize an image file on the fly using fly-dynamic-image-resizer
3.16.9.2. post-template.php wp:t:post
- Only
get_the_*( $post )can pass a different post or post id to return values. Useesc_html_e()to echoget_the_guid( $post = 0)-
if (! is_admin() )
- wp:filter:protected_title_format
- add
Protected: - wp:filter:private_title_format
- add
Private:$post->post_title- wp:filter:the_title
get_the_content( $more_link_text = null, $strip_teaser = false, $post = null )get_the_excerpt( $post = null )get_the_password_form( $post = 0 )
- ID of current item or false
the_ID()- echo
the_*()always uses theglobal $postto echothe_title( $before = '', $after = '', $echo = true )- return or echo
$before . get_the_title() . $after the_title_attribute( string|array $args = '' )similar to above but tags are stripped and escaped for HTML attributes
$defaults = [ 'before' => '', 'after' => '', 'echo' => true, 'post' => get_post(), ];
the_content( $more_link_text = null, $strip_teaser = false)- use wp:filter:the_content
- (no term)
the_excerpt()- When another post/post id is passed on
$excerpt = apply_filters('the_excerpt', get_post_field('post_excerpt', $item->id));
get_post_class()- e.g.
post-123 myposttype type-myposttype status-publish has-post-thumbnail - filter
post_class($classes, $class, $post->ID)
- e.g.
wp_list_pages( $args = '' )- $args
- post_type
- default
page. For CPT,wp_list_pages( array('post_type' => 'articles' ) ) - echo
- default 1, to echo. Set 0 to return html string
- title_li
- default
__( 'Pages' ). Add a<li>as title before any child post elements - child_of
- default 0 (return child posts of any parent posts). specify the parent post ID
- sort_column
- default
'menu_order, post_title'
- $args
- get_the_excerpt, the_excerpt
- Use
the_excerpt()echoes current post and it callsget_the_excerpt( $post = null )- If no excerpt is set in UI, it pulls the first 55 words. If it's set, it displays the whole manually set text
- When used outside a loop, refer to wp:loop:outside
3.16.9.3. category-template.php wp:t:category
- Refer to wp:post
- display taxonomy terms in a list
wp_list_categories( $args = '')- taxonomy
- default 'category'. For custom taxonomy,
wp_list_categories( [ 'taxonomy' => 'mycats' ] ) - child_of
- default 0. Specify the parent term id
3.16.9.4. general-template wp:t:general
// In template <?php wp_head(); ?> // In plugin add_action('wp_head', 'lili_wphead_dfp'); function lili_wphead_dfp() { // add inline script to header $o = "<script>...</script>"; echo $o; } // Load with dependancy (jQuery is loaded) function lili_wphead_dfp() { if ( wp_script_is( 'jquery', 'done' ) ) { ?> <script type="text/javascript"> // jQuery code </script> <?php } }
- get_header, get_footer, get_sidebar, get_search_form, comments_template
- wp:f:get_header
get_header( $name = null )- wp:f:get_footer
get_footer( $name = null )- wp:f:get_sidebar
get_sidebar( $name = null )Refer to wp:f:dynamic_sidebar$templates[] = "sidebar-{$name}.php"; $templates[] = 'sidebar.php';locate_template( $templates, true );
- wp:f:get_search_form
get_search_form( $args = array() )- (no term)
- wp:f:comments_template
- (no term)
These functions are to load template files
- System default load paths
wp-includes/theme-compat/header.php, footer.php, sidebar.php
header-{name}.phpfooter-{name}.phpsidebar-{name}.php{slug}-{name}.phpsearchform.phpcomment.php
get_template_part( string $slug, string $name = null ):: wp:f:get_template_part
requirefile{slug.php}or{slug}-{name}.php<?php get_template_part( 'template-parts/header/header', 'image' ); ?> // require( 'path-to-theme/template-parts/header/header-image.php' );
- Call
locate_template- Call
load_template
- Call
- wp:action:get_template_part$slug
Pass variables to template
// parent $args = ['...']; set_query_var( 'my_query_args', $args ); get_template_part('template-parts/test'); // test.php $query = new WP_Query( $my_query_args );
get_bloginfo( $show = '', $filter = 'raw' )- retrieves info about the current site
https://developer.wordpress.org/reference/functions/get_bloginfo/
<img src="<?php bloginfo('template_url'); ?>/assets/images/a.jpg" alt="" />
$filter- version
- '4.9.4'
- template_url
- URL of active theme's directory
- url or wpurl
- Site address URL Settings > General
bloginfo( $show ='' )echo get_bloginfo( $show, 'display' )
- wp_editor wp:f:wp_editor
- Render an editor for field in a page in the typical fashion used in Posts and Pages
- Doc
- (no term)
- May be used in
- (no term)
- Use wp:f:update_post_meta in wp:action:save_post for storing the new field
- has_custom_logo, get_custom_logo, the_custom_logo wp:custom logo:display
get_custom_logo returns markup the_custom_logo displays markup
$custom_logo_id = get_theme_mod( 'custom_logo' ); $logo = wp_get_attachment_image_src( $custom_logo_id , 'full' ); if ( has_custom_logo() ) { echo '<img src="'. esc_url( $logo[0] ) .'">'; } else { echo '<h1>'. get_bloginfo( 'name' ) .'</h1>'; }
3.16.9.5. author-template.php
get_the_author_meta( string $field = '', int|false $user_id = false ) : stringwp:f:get_the_author_meta
- $field
- can be
- display_name
- $user_id
- default false to get the current post's user. Can be
$post->post_author
3.16.9.6. bookmark_template.php
3.16.9.7. comment-template.php
wp_list_comments( $args = array(), $comments = null)- wp:f:wp_list_comments
- List comments in
comments.phptemplate. Use wp:filter:wp_list_comments_args
- List comments in
comment_form( $args = array(), $post_id = null)- outputs comment form wp:f:comment_form
comments_template( $file = '/comments.php', $separate_comments = false )- wp:f:comments_template
- Default loads
wp-includes/theme-compat/comments.php
- Default loads
3.16.9.8. link-template.php
get_permalink( int|WP_Post $post = 0, bool $leavename = false)- get full permalink for current post or post ID
- $post
- optional, default
global $post - $leavename
- whether to keep post/page name
- return string|false
- (no term)
- Applied filters
the_permalink( int|WP_Post $post = 0 )- print escaped
get_permalink- Applied filters
- wp:filters:the_permalink
edit_post_link( $text = null, $before = '', $after = '', $id = 0, $class = 'post-edit-link' )- print edit post link
edit_post_link( __( '<span class="small">(Edit)</span>', 'genesis' ) );
get_theme_file_uri( $file = '' )- wp:f:get_theme_file_uri
Check for file existance and returns URL. Get from child theme first js/my-scripts.js. Fall back to parent theme
wp_enqueue_script( 'my-script', get_theme_file_uri( 'js/my-script.js' ) );
get_theme_file_path( $file = '' )- wp:f:get_theme_file_path
Return the path in filesystem so that you can use
filemtimewp_enqueue_script( 'my-script', get_theme_file_uri( 'js/my-script.js' ), array(), filemtime( get_theme_file_path( 'js/my-script.js' ) ) );
3.16.9.9. media-template.php
3.16.9.10. nav-menu-template.php
3.16.9.11. feed.php
the_title_rss()- echo
get_the_title_rss()wp:f:the_title_rss - (no term)
get_the_title_rss()get_the_title(): wp:f:get_the_title- wp:filter:the_title_rss
3.16.10. Post API wp-includes/post.php wp:api:post
3.16.10.1. get_post( int|WP_Post|null $post = null, string $output = OBJECT, string $filter = 'raw' ) wp:f:get_post
- Default is
global $post - Return wp:post object
3.16.10.2. get_metadata( string $meta_type, int $object_id, string $meta_key = '', bool $single = false ) wp:f:get_metadata
$meta_key- default to return data for all keys
$single- wheter to return single value. return array if it's false
- (no term)
e.g. Return post thumbnail alt text
// return post thumbnail alt text $thumb_id = get_post_thumbnail_id(get_the_ID()); $alt = get_post_meta($thumb_id, '_wp_attachment_image_alt', true); if( $alt ): echo $alt; endif; // return a post metadata $meta = get_post_meta(get_the_ID()); echo $meta['header-title'][0];
3.16.10.3. get_post_meta( int $post_id, string $key = '', bool $single = false ) wp:f:get_post_meta
return get_metadata( 'post', $post_id, $key, $single );- refer to wp:f:get_metadata
3.16.10.4. get_posts( $args = null ) : WP_Post[]|int[] wp:f:get_posts
// new WP_Query(); is created $args = array( 'post_type' => 'speaker', 'posts_per_page' => -1, ); $r = get_posts($args); // run loop // reset global $post wp_reset_postdata();
3.16.10.5. wp_get_post_terms( $post_id = 0, $taxonomy = 'post_tag', $args = array() ) : WP_Term[]|WP_Error wp:f:wp_get_post_terms
- Retrieve terms for a post
$taxonomy- default only return terms that are tags
$post_id- default current post if it's in a loop
$args- default
$args = array('orderby' => 'name', 'order' => 'ASC', 'fields' => 'all'); - Return
- an array of taxonomy terms as objects, empty array or WP_Error
wp_get_post_terms( $post_id, 'publication', ['field' => 'slugs'] )Sample
Array ( [0] => WP_Term Object ( [term_id] => 145 [name] => Example Category [slug] => example-cat [term_group] => 0 [term_taxonomy_id] => 145 [taxonomy] => adcpt_categories [description] => [parent] => 0 [count] => 2 [filter] => raw ) )Return all terms for a post
global $post; $tax = get_taxonomies(); $terms = wp_get_post_terms($post->ID, $tax);
3.16.10.6. add_post_type_support( string $post_type, string|array $supports) wp:f:add_post_type_support
remove_post_type_support( string $post_type, string $supports )post_type_supports( string $post_type, string $feature )- string
string/array
- title
- content
- author
- current theme must support Post Thumbnails
- excerpt
- trackbacks
- wp:Custom_Fields. Don't have to add 'custom-fields' in order to use wp:plugin:acf
- comments
- revisions
- menu order, hierarchical must be true
- add post formats
add_action('init', 'lili_custom_post_type'); function lili_custom_post_type() { $labels = [ // array, wp:register_post_type:labels ]; $args = [ 'label' => 'Plural Name', // string. required. 'labels' => $labels, 'description' => '', // string, optional 'public' => false, /* Default. bool. * exclude_from_search, publicly_queryable, show_in_nav_menus, show_ui * true: false, true, true, true * false: true, false, false, false */ 'exclude_from_search' => false, // default. optional. 'site/?s=search-term' 'publicly_queryable' => true, // optional. take 'public' value. /* ?post_type={post_type_key}, * ?{post_type_key}={single_post_slug}, * ?{post_type_query_var}={single_post_slug} */ 'show_ui' => true, // optional. bool. default take 'public' value. 'show_in_nav_menus' => true, // optional. bool. inherits 'public' value. 'show_in_menu' => true, // optional, bool or string. default inherits show_ui // false :: do not display in admin menu // true :: display as a top level menu // 'some string' :: display as a submenu of a menu 'some string' 'show_in_admin_bar' => true, // optional, bool, default inherits show_in_ 'supports' => ['title', 'editor'], // optional. array/bool. wp:f:add_post_type_support ]; register_post_type('movies', $args ); }
Use wp:action:pre_get_posts to enable query for this new post type
3.16.10.7. register_post_type wp:f:register_post_type wp:add cpt
- https://codex.wordpress.org/Function_Reference/register_post_type
- Plugins use this function
- Called actions
- wp:action:registered_post_type
- Called functions
- wp:action:pre_get_posts
- Refresh Permalink from UI or
flush_rewrite_rules()in plugin/theme activation and deactivation hooks - Do not call before wp:action:init
register_post_type( string $post_type, array|string $args = array() ) : WP_Post_Type|WP_Error- $post_type
- string. < 20 characters. All lowercase and no space
- $args
- array of arguments
- query_var
- default true to use
$post_typeas public query_var wp:f:register_post_type:query_var
add_action('init', 'lili_custom_post_type'); function lili_custom_post_type() { $labels = [ // array, wp:register_post_type:labels ]; $args = [ 'label' => 'Plural Name', // string, optional, default labels['name'] 'labels' => $labels, 'description' => '', // string, optional 'public' => false, /* Default false * exclude_from_search, publicly_queryable, show_in_nav_menus, show_ui * true: false, true, true, true * false: true, false, false, false */ 'exclude_from_search' => false, // default. optional. 'site/?s=search-term' 'publicly_queryable' => true, // optional. take 'public' value. /* ?post_type={post_type_key}, * ?{post_type_key}={single_post_slug}, * ?{post_type_query_var}={single_post_slug} */ // 'query_var' => true, // bool|string d:true to use $post_type. Set to false means the post type cannot be loaded at /?{query_var}={single_post_slug}. It has no effect when public_queryable is false 'show_ui' => true, // optional. bool. default take 'public' value. 'show_in_nav_menus' => true, // optional. bool. inherits 'public' value. 'show_in_menu' => true, // optional, bool or string. default inherits show_ui // false :: do not display in admin menu // true :: display as a top level menu // 'some string' :: display as a submenu of a menu 'some string' 'show_in_admin_bar' => true, // optional, bool, default inherits show_in_menu 'capability_type' => 'post', // default, for naming capabilities e.g. `edit_published_posts` 'map_meta_cap' => false, // default false. If you specify capability_type other than post, you should set this to true. Optionally, build map_meta_cap hooks to give further granular meta capabilities 'supports' => ['title', 'editor'], // optional. array/bool. wp:f:add_post_type_support // 'custom-fields' :: don't have to add in order to use wp:plugin:acf // 'title', 'editor' (content) this is the default // 'author','excerpt', 'trackbacks', // 'thumbnail' (featured image, current theme must also support post-thumbnails) // 'comments' (also will see comment count balloon on edit screen) // 'revisions' (will store revisions) // 'page-attributes' (menu order, hierarchical must be true to show Parent option) // 'post-formats' add post formats, see Post Formats // 'hierarchical' => true // default false, when true 'supports' should contain 'page-attributes' 'has_archive' => false, // d:false 'rewrite' => [ 'slug' => 'movies', // d:$post_type 'with_front' => true, // d:true should the permalink strucutre to be prepended with the front base 'feeds' => false, // d:$has_archive should a feed permalink structure be built 'pages' => true, // d:true should the permalink structure provide for pagination 'ep_mask' => EP_PERMALINK, // don't know what it is.. ], // bool|array d:true and use $post_type as slug ]; register_post_type('movies', $args); } add_action( 'pre_get_posts', 'lili_pre_get_posts' ); // Should not call this in template because in template the $query is already run function lili_pre_get_posts( $query ) { if ( ! is_page() && ! is_home() && $query->is_main_query() ) { $query->set( 'post_type', array( 'post', 'movies' ) ); } // this function should return nothing. }
Get Custom Post Types
$cpts = get_post_types( array( 'public' => true, '_builtin' => false, ));
- labels wp:register_post_type:labels
'labels' => array( 'name' => __( 'Jobs' ), 'singular_name' => __( 'Job' ) )
- menu_icon
optional, default null > posts icon 'dashicons-video-alt' (Uses the video icon from Dashicons) 'get_template_directory_uri() . "/images/cutom-posttype-icon.png"' (Use a image located in the current theme) 'data:image/svg+xml;base64,' . base64_encode( "<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="20px" height="20px" viewBox="0 0 459 459"> <path fill="black" d="POINTS"/></svg>" )' (directly embedding a svg with 'fill="black"' will allow correct colors.)
3.16.10.8. register_post_status wp:f:register_post_status
- Register post status
- Do not use before wp:action:init
- add to wp:global:wp_post_statuses
- Built-in post statuses
_builtin- publish
- public
- future
- protected
- draft
- protected
- pending
- protected
- private
- private
- trash
- internal
- auto-draft
- internal
- inherit
- internal Used with child post such as Attachments (e.g. featured image) and Revisions
- request-pending
- internal
- request-confirmed
- internal
- request-failed
- internal
- request-completed
- internal
- Extra post statuses
- wp:plugin:acf
- acf-disabled
register_post_status( string $post_status, array|string $args = array() )$args
$args = [ 'label' => 'Post status name for translation', // bool|string d:$post_status 'label_count' => '', // bool|array d:none text to display on admin screen e.g. _n_noop( 'Unread <span class="count">(%s)</span>', 'Unread <span class="count">(%s)</span>' ) 'exclude_from_search' => false, // bool d:$internal whether to exclude posts with this post status 'internal' => false, // bool d:false whether the status is for internal use only 'public' => false, // bool whether posts of this status should be shown in the front end of the site // '_builtin' => false, // bool d:false core-use only 'protected' => false, // bool d:false whether posts with this status should be protected 'private' => false, // bool d:false whether posts with this status should be private 'publicly_queryable' => false, // bool d:$public whether posts with this status should be publicly-queryable 'show_in_admin_all_list' => false, // bool d:$internal whether to include posts in the edit listing for their post type 'show_in_admin_status_list'=> false, // bool d:$internal show in the list of statuses with post counts at the top of the edit listings e.g. All (12) | Published (9) | My Custom Status (2) ];
function my_custom_post_status(){ register_post_status( 'unread', array( 'label' => _x( 'Unread', 'post' ), 'public' => true, 'exclude_from_search' => false, 'show_in_admin_all_list' => true, 'show_in_admin_status_list' => true, 'label_count' => _n_noop( 'Unread <span class="count">(%s)</span>', 'Unread <span class="count">(%s)</span>' ), ) ); } add_action( 'init', 'my_custom_post_status' );
3.16.10.9. register_post_meta wp:f:register_post_meta
register_post_meta( $post_type, $meta_key, array $args ) : boolregister_meta( 'post', $meta_key, $args )
3.16.10.10. update_post_meta wp:f:update_post_meta
3.16.11. Post Formats wp-includes/post-formats.php
3.16.11.1. has_post_format wp:f:has_post_format
In template
if ( has_post_format( 'aside' )) { // code to display the aside format post here } else if (has_post_format('gallery')) { // stuff to display the gallery format post here } else if (has_post_format('link')) { // stuff to display the link format post here }else { // code to display the normal format post here }
3.16.12. Rewrite API wp:api:rewrite
3.16.12.1. add_rewrite_rule, add_rewrite_tag
- wp:f:add_rewrite_rule
- Need to flush
- Settings > Permalinks > click Save Changes without any changes to add into wp:options:rewrite_rules
- (no term)
- Add a rewrite rule that transforms a URL structure to a set of query vars
- (no term)
- Use with wp:f:add_rewrite_tag
- (no term)
- wp:action:init
- (no term)
add_rewrite_rule( string $regex, string|array $query, string $after = 'bottom' )- $query
$matches[]to retrieve the values of a matched URL, capture group data starts at 1, not 0- $after
- This can either be 'top' or 'bottom'. 'top' will take precedence over WordPress's existing rules, where 'bottom' will check all other rules match first
- (no term)
Sample
function custom_rewrite_basic() { add_rewrite_rule('^leaf/([0-9]+)/?', 'index.php?page_id=$matches[1]', 'top'); } add_action('init', 'custom_rewrite_basic');
- wp:f:add_rewrite_tag
- add a query var wp:filter:query_vars
Sample
function custom_rewrite_tag() { add_rewrite_tag('%food%', '([^&]+)'); add_rewrite_tag('%variety%', '([^&]+)'); } add_action('init', 'custom_rewrite_tag', 10, 0); function custom_rewrite_rule() { add_rewrite_rule('^nutrition/([^/]*)/([^/]*)/?','index.php?page_id=12&food=$matches[1]&variety=$matches[2]','top'); } add_action('init', 'custom_rewrite_rule', 10, 0);
Create a page called Nutrition with page id 12 and use a custom template my-custom-template.php
/** * Template Name: Nutritional Information */ get_header(); global $wp_query; echo 'Food : ' . $wp_query->query_vars['food']; echo '<br />'; echo 'Variety : ' . $wp_query->query_vars['variety']; // ... more ... get_footer();
3.16.12.2. add_feed( string $feedname, callback $function ) : string wp:f:add_feed wp:rss wp:feed
- Default template
/wp-includes/feed-rss2.php- Functions to use in The Loop (template)
- wp:api:feed
- add action
- wp:action:do_feed_$feedname which does the callback
$functionat p:10 - return the action name
do_feed_$feedname- (no term)
- Refer to wp:f:do_feed
function lili_RSS() { add_feed('lili-editorial', 'lili_editorial_RSS'); } add_action('init', 'lili_RSS'); function lili_editorial_RSS() { get_template_part('rss','feedname'); // rss is slug and feedname is an extra name // Search in order // /themes/child/rss-feedname.php // /themes/parent/rss-feedname.php // /themes/child/rss.php // /themes/parent/rss.php // the feed url is http://a.ca/feed/lili-editorial which is the feed name defined in add_feed }
3.16.13. Feed API wp-includes/feed.php wp:api:feed
- Refer to wp:f:add_feed
<?php $args = array( 'post_type' => array('blog'), // e.g. any 'tag' => 'editors-choice', 'orderby' => 'date', // Default. e.g. title 'order' => 'DESC', // Default. e.g. ASC ); ?> query_posts($args); <?php while (have_posts()) : the_post(); ?> <item> <title><?php the_title_rss(); ?></title> <?php if (has_post_thumbnail($post->ID)) : ?> <?php $image = wp_get_attachment_image_src( get_post_thumbnail_id( $post->ID ), 'thumbnail' ); ?> <feature_image><?php echo $image[0]; ?></feature_image> <?php endif; ?> <pubDate><?php echo mysql2date('D, d M Y H:i:s +0000', get_post_time('Y-m-d H:i:s', true), false); ?></pubDate> <?php // Text Version ?> <excerpt><![CDATA[<<?php echo get_the_exerpt(); ?>]]></excerpt> <excerpt_html><![CDATA[<<?php echo the_exerpt_rss(); ?>]]></excerpt_html> <?php echo the_category_rss('rss2'); // categories and tags ?> </item> <?php endwhile; ?>
3.16.14. Dependency API wp-includes/script-loader.php wp:api:dependency
3.16.14.1. wp_localize_script wp:f:wp_localize_script
- Must be called after the script has been registered using
wp_register_script()orwp_enqueue_script() - Use to define variables as an object
// register a script that may be loaded using wp_enqueue_script() // wp_register_script( 'unique_handle', 'path/to/myscript.js' ); // later // wp_enqueue_script( 'unique_handle' ); // or you enque directly wp:f:wp_enqueue_script wp_enqueue_script( 'unique_handle', get_theme_file_uri( '/path/to/myscript.js' ) ); // Pass vars in $translation_array to file path/to/myscript.js (registered using handle `unique_handle`) $translation_array = array( 'some_string' => 'hello'; 'a_value' => '10' ); wp_localize_script( 'unique_handle', 'object_name', $translation_array ); wp_enqueue_script( 'unique_handle' );
In the enqueued javascript file path/to/myscript.js
console.log( object_name.some_string ); // default all values are strings parseInt( object_name.some_int, 10 ); // pass parameter as int
If you just want to define pass PHP variables and don't want to create empty js files
wp_localize_script( 'jquery', 'mynamesapce', array('ajaxurl' => admin_url( 'admin-ajax.php' ) ) );
wp_enqueue_script( 'twentyseventeen-skip-link-focus-fix', get_theme_file_uri( '/assets/js/skip-link-focus-fix.js' ), array(), '1.0', true ); wp_localize_script( 'twentyseventeen-skip-link-focus-fix', 'twentyseventeenScreenReaderText', $twentyseventeen_l10n );
3.16.14.2. wp_enqueue_script, wp_script_add_data wp:f:wp_enqueue_script
wp_enqueue_script( string $handle, string $src = '', array $deps = array(), string|bool|null $ver = false, bool $in_footer = false ) wp_enqueue_script( 'html5', get_theme_file_uri( '/assets/js/html5.js' ), array(), '3.7.3' ); wp_script_add_data( 'html5', 'conditional', 'lt IE 9' );
When a script is added with dependency on jQuery, jQuery is auto loaded by wp. Change jQuery
wp_deregister_script('jquery'); wp_enqueue_script('jquery', 'https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js', array(), null, true); wp_enqueue_script('my-custom-script', get_template_directory_uri() .'/js/my-custom-script.js', array('jquery'), null, true);
If there's script depends on jquery and it's loaded in header, even though jquery is set to load in footer, jquery will be loaded in header.
3.16.14.3. wp_enqueue_style, wp_style_add_data wp:f:wp_enqueue_style wp:f:wp_style_add_data
- Used in wp:action:wp_enqueue_scripts
- Use wp:filter:script_loader_src
wp_enqueue_style( string $handle, string $src = '', array $deps = array(), string|bool|null $ver = false, string $media = 'all' )- (array) (Optional) An array of registered stylesheet handles this stylesheet depends on
- Default value: array()
- (string|bool|null) (Optional) String specifying stylesheet version number, if it has one, which is added to the URL as a query string for cache busting purposes. If version is set to false, a version number is automatically added equal to current installed WordPress version. If set to null, no version is added
- Default value: false
- (string) (Optional) The media for which this stylesheet has been defined. Accepts media types like 'all', 'print' and 'screen', or media queries like '(orientation: portrait)' and '(max-width: 640px)'
- Default value: 'all'
function wpb_add_google_fonts() { wp_enqueue_style( 'wpb-google-fonts', 'http://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,700,300', false ); } add_action( 'wp_enqueue_scripts', 'wpb_add_google_fonts' );
Conditional statements are removed from IE 10 or IE 11. Refer to css:font-face
// Load the main stylesheet wp_enqueue_style( 'my-theme', get_stylesheet_uri() ); /** * Load our IE-only stylesheet for all versions of IE: * <!--[if IE]> ... <![endif]--> */ wp_enqueue_style( 'my-theme-ie', get_stylesheet_directory_uri() . "/css/ie.css", array( 'my-theme' ) ); wp_style_add_data( 'my-theme-ie', 'conditional', 'IE' ); /** * Load our IE version-specific stylesheet: * <!--[if IE 7]> ... <![endif]--> */ wp_enqueue_style( 'my-theme-ie7', get_stylesheet_directory_uri() . "/css/ie7.css", array( 'my-theme' ) ); wp_style_add_data( 'my-theme-ie7', 'conditional', 'IE 7' ); /** * Load our IE specific stylesheet for a range of older versions: * <!--[if lt IE 9]> ... <![endif]--> * <!--[if lte IE 8]> ... <![endif]--> * NOTE: You can use the 'less than' or the 'less than or equal to' syntax here interchangeably. */ wp_enqueue_style( 'my-theme-old-ie', get_stylesheet_directory_uri() . "/css/old-ie.css", array( 'my-theme' ) ); wp_style_add_data( 'my-theme-old-ie', 'conditional', 'lt IE 9' ); /** * Load our IE specific stylesheet for a range of newer versions: * <!--[if gt IE 8]> ... <![endif]--> * <!--[if gte IE 9]> ... <![endif]--> * NOTE: You can use the 'greater than' or the 'greater than or equal to' syntax here interchangeably. */ wp_enqueue_style( 'my-theme-new-ie', get_stylesheet_directory_uri() . "/css/new-ie.css", array( 'my-theme' ) ); wp_style_add_data( 'my-theme-ie', 'conditional', 'gt IE 8' );
3.16.15. Taxonomy API wp-includes/taxonomy.php wp:api:taxonomy
3.16.15.1. get_term_by( $field, $value, $taxonomy = '', $output = OBJECT, $filter = 'raw' )
- Return WP_Term|array|false
get_term_by( 'slug', 'my-term-slug', 'taxonomy-of-the-term' )get_term_by( 'slug', get_query_var( 'term' ), get_query_var( 'taxonomy' ) )- slug, name, id or term_taxonomy_id
- OBJECT, ARRAY_A, ARRAY_N
3.16.15.2. register_taxonomy( string $taxonomy, array|string $object_type, array|string $args = array() ) wp:f:register_taxonomy
- https://codex.wordpress.org/Function_Reference/register_taxonomy
- Do not call before wp:action:init
- Parameters
- $taxonomy
- tax slug <= 32 characters
- $object_type
- e.g. post type slug
- (no term)
$args
- query_var
- d:$taxonomy wp:f:register_taxonomy:query_var
- update_count_callback
- wp:f:register_taxonomy:update_count_callback
- d:''
- affect wp:f:wp_update_term_count_now
- wp:global:wp_taxonomies
- wp:action:registered_taxonomy
add_action( 'init', 'create_topics_hierarchical_taxonomy', 0 ); //create a custom taxonomy name it topics for your posts function create_topics_hierarchical_taxonomy() { // Add new taxonomy, make it hierarchical like categories //first do the translations part for GUI $labels = array( 'name' => _x( 'Topics', 'taxonomy general name' ), 'singular_name' => _x( 'Topic', 'taxonomy singular name' ), 'search_items' => __( 'Search Topics' ), 'all_items' => __( 'All Topics' ), 'parent_item' => __( 'Parent Topic' ), 'parent_item_colon' => __( 'Parent Topic:' ), 'edit_item' => __( 'Edit Topic' ), 'update_item' => __( 'Update Topic' ), 'add_new_item' => __( 'Add New Topic' ), 'new_item_name' => __( 'New Topic Name' ), 'menu_name' => __( 'Topics' ), ); // Now register the taxonomy register_taxonomy('topics',array('post'), array( 'hierarchical' => true, 'labels' => $labels, 'show_ui' => true, 'show_admin_column' => true, 'query_var' => true, 'rewrite' => array( 'slug' => 'topic' ), )); }
Non-hierarchical taxonomy
add_action( 'init', 'create_topics_nonhierarchical_taxonomy', 0 ); function create_topics_nonhierarchical_taxonomy() { // Labels part for the GUI $labels = array( 'name' => _x( 'Topics', 'taxonomy general name' ), 'singular_name' => _x( 'Topic', 'taxonomy singular name' ), 'search_items' => __( 'Search Topics' ), 'popular_items' => __( 'Popular Topics' ), 'all_items' => __( 'All Topics' ), 'parent_item' => null, 'parent_item_colon' => null, 'edit_item' => __( 'Edit Topic' ), 'update_item' => __( 'Update Topic' ), 'add_new_item' => __( 'Add New Topic' ), 'new_item_name' => __( 'New Topic Name' ), 'separate_items_with_commas' => __( 'Separate topics with commas' ), 'add_or_remove_items' => __( 'Add or remove topics' ), 'choose_from_most_used' => __( 'Choose from the most used topics' ), 'menu_name' => __( 'Topics' ), ); // Now register the non-hierarchical taxonomy like tag register_taxonomy('topics','post',array( 'hierarchical' => false, 'labels' => $labels, 'show_ui' => true, 'show_admin_column' => true, 'update_count_callback' => '_update_post_term_count', 'query_var' => true, 'rewrite' => array( 'slug' => 'topic' ), )); }
3.16.15.3. wp_update_term_count_now( array $terms, string $taxonomy ) wp:f:wp_update_term_count_now
- $terms
- array of term_taxonomy_id
- return true
- when complete
- (no term)
- It's only called when
- a post's post status is updated (update count for the post's associated terms)
- a tax is added or removed (update count for that tax only)
- Not cleared when cache is cleared and has to be run manually to update count
call_user_func( $taxonomy->update_count_callback, $terms, $taxonomy )- if wp:f:register_taxonomy:update_count_callback is defined
_update_post_term_count( $terms, $taxonomy )- if there's any attached post types that exist, get total count of publish posts of attached post types
- wp:action:edit_term_taxonomy
- Update wp:db:wp_term_taxonomy:count
- wp:action:edited_term_taxonomy
_update_generic_term_count( $terms, $taxonomy )- if there's no attached post type that exists, get total count of anything from wp:db:wp_term_relationships
3.16.15.4. wp:action:registered_taxonomy
- Add action before wp:f:register_taxonomy is called
3.16.15.5. Add a sortable column to taxonomy edit page
- Taxonomy
- sponsor_type
- (no term)
- Refer to wp:api:plugin for adding a column to custom post type
- Filter
manage_${taxonomy}_custom_column - wp:api:plugin:filter:managetaxonomy_custom_column
<?php add_filter( 'manage_edit-sponsor_type_columns', function ( $columns ) { $columns['my_term_id'] = __( 'Term ID' ); // add a column $columns['sponsorship_type_order'] = __( 'Order' ); // add a column // // to see existing columns, wp-admin/edit-tags.php?taxonomy=sponsor_type //var_dump('hello'); var_dump($columns); //unset( $columns['cpt-shows'] ); // Don't allow sort by a column return $columns; } ); // populate values in column Order add_filter( 'manage_sponsor_type_custom_column', function ( $value, $column_name, $term_id ) { $term = get_term( $term_id, 'sponsor_type' ); $order = get_field( 'sponsorship_type_order', $term ); switch ( $column_name ) { case 'sponsorship_type_order': $value = $order; break; default: break; } return $value; }, 10, 3 ); // make the column sortable add_filter( 'manage_edit-sponsor_type_sortable_columns', function ( $columns ) { // see existing sortable columns //var_dump('hello'); var_dump($columns); $columns['sponsorship_type_order'] = 'custom_order'; // don't use order or orderby as the url parameter name return $columns; } ); // add sort mechanism add_filter( 'terms_clauses', function ( $pieces, $taxonomies, $args ) { global $pagenow; if ( ! is_admin() ) { return $pieces; } if ( is_admin() && $pagenow == 'edit-tags.php' && $taxonomies[0] == 'sponsor_type' && ( ! isset( $_GET['orderby'] ) || $_GET['orderby'] == 'custom_order' ) ) { // to further customize the sorting. By default, using the populated value to sort is good enough // e.g. If orderby is not set, add some default from wp_options // $pieces['join'] .= " INNER JOIN wp_options AS opt ON opt.option_name = concat('issue_',t.term_id,'_issue_date')"; // $pieces['orderby'] = "ORDER BY opt.option_value"; // $pieces['order'] = isset($_GET['order']) ? $_GET['order'] : "DESC"; } return $pieces; }, 10, 3 );
3.16.15.6. get_taxonomies( $args = array(), $output = 'names', $operator = 'and' ) : string[]|WP_Taxonomy[] wp:f:get_taxonomies
// @return string[]|WP_Taxonomy[] An array of taxonomy names or objects. $return = [ 'category' => 'category', 'post_tag' => 'post_tag', 'custom_taxonomy_name' => 'custom_taxonomy_name', // ... ];
3.16.16. Shortcode API wp-includes/shortcodes.php wp:api:shortcode
3.16.16.1. add_shortcode wp:f:add_shortcode
// [bar-tag foo="foo-value"] // Use underscore instead of hyphen e.g. bar_tag function bartag_func( $atts, $content, $tag ) { $a = shortcode_atts( array( 'class' => 'something', // default value 'attr_2' => 'something else', // 2nd attribute ), $atts ); // get current post // global $post; $post->ID // Use return not echo return '<span class="'.esc_attr($a['class']).'">' .$content // or // .do_shortcode($content) ."foo = {$a['foo']}" .'</span>'; /* ob_start(); ?> <HTML> <here> ... <?php return ob_get_clean(); */ /* ob_start(); get_template_part( 'template-parts/form/contact-us'); return ob_get_clean(); */ } add_shortcode( 'bar-tag', 'bartag_func' ); // In order to run shortcode in Widgets, add this add_filter('widget_text','do_shortcode'); // wp:filter:widget_text
Pass array to shortcode attribute
$data = [ [],[],[] ]; var_dump(serialize($data)); // use single quotes // [your-shortcode data='serialized-string'] function your_shortcode_cb() { // ... $data = unserialize($a['data']); }
3.16.16.2. shortcode_unautop wp:f:shortcode_unautop
- Used in wp:filter:the_content
- Ensure shortcodes are not wrapped in
<p>...</p> - Refer to
add_filter( 'the_content', 'shortcode_unautop' );
3.16.17. OEmbed API wp-includes/embed.php wp:api:oembed
3.16.17.1. Basics
- Shortcode
[embed width="560" height="315"]https://youtu.be/abc[/embed]- Whitelist of sources
- https://wordpress.org/support/article/embeds/
- Embeds that aren't associated with a post will be cached in oembed_cache post type
- Embeds that are associated with a post is stored in the post's metadata
3.16.18. Media API wp-includes/media.php wp:api:media
3.16.18.1. add_image_size wp:f:add_image_size
- Use wp:action:after_setup_theme
add_image_size( string $name, int $width, int $height, bool|array $crop = false )- Every newly uploaded image will have image files with all defined image sizes
$cropcan be- false
- will be scaled
- true
- will be cropped to the specified dimensions using center positions
array( $x_crop_position, $y_crop_position )- will be cropped to the specified dimensions within the defined crop area
- $x_crop_position
- 'left', 'center' or 'right'
- $y_crop_position
- 'top', 'center' or 'bottom'
- If the original image (e.g. featured image) is smaller than the thumbnail image size, by default, neither it's set to crop to scale, the result thumbnail will take the original image without scaling up
Refer to wp:filter:image_size_names_choose
function lili_theme_setup() { add_image_size('lili-w700', 700, 9999); // fixed width, unlimited height add_image_size('lili-w700h600', 700, 600, true); // crop in exact dimension }
Force to scale up the thumbnail if original image is smaller than the thumbnail size, and maintain ratio
- Does not seem to work for scale (non-croppable) image sizes last time I tried.. Ratio is not maintained for scaling image sizes
function alx_thumbnail_upscale( $default, $orig_w, $orig_h, $new_w, $new_h, $crop ){ // comment the line below to enable upscale for both scale and crop if ( !$crop ) return null; // let the wordpress default function handle automatic scale $aspect_ratio = $orig_w / $orig_h; $size_ratio = max($new_w / $orig_w, $new_h / $orig_h); $crop_w = round($new_w / $size_ratio); $crop_h = round($new_h / $size_ratio); $s_x = floor( ($orig_w - $crop_w) / 2 ); $s_y = floor( ($orig_h - $crop_h) / 2 ); return array( 0, 0, (int) $s_x, (int) $s_y, (int) $new_w, (int) $new_h, (int) $crop_w, (int) $crop_h ); } add_filter( 'image_resize_dimensions', 'alx_thumbnail_upscale', 10, 6 );
// after this call, my-image-size-style-name thumbnail should be ready // by default, wp does not scale up if the original file is smaller than the requested thumbnail image size // regardless of the add_image_size setting // Original my-image-size-style-name setting: add_image_size( 'my-image-size-style-name', 340, 200, true ); regenerateImageSizeIfOneIsMissing( get_post_thumbnail_id( $post->ID ), 'my-image-size-style-name' ); $image = wp_get_attachment_image_src( get_post_thumbnail_id( $post->ID ), 'my-image-size-style-name' ); // regenerate all image sizes for an attachment (e.g. featured image) if the specified image size does not exist function regenerateImageSizeIfOneIsMissing( $attachment_id, $check_image_size, $enforce_enlarge = false ) { global $_wp_additional_image_sizes; // stop if no file id is provided or the image size is not defined if ( ! $attachment_id || ! isset( $_wp_additional_image_sizes[ $check_image_size ] ) ) { return false; } //print_r($attachment_id); //print '<pre>'; print_r( $_wp_additional_image_sizes ); print '</pre>'; $metadata = wp_get_attachment_metadata( $attachment_id ); if ( ! ( isset( $metadata['sizes'] ) && isset( $metadata['sizes'][ $check_image_size ] ) ) ) { //printf('Image size "%s" does not exist, regenerate all image sizes for this attachment', $check_image_size); $fullsizepath = get_attached_file( $attachment_id ); if ( false === $fullsizepath || ! file_exists( $fullsizepath ) ) { // stop if there is no full file or it does not exist return false; } if ( ! function_exists( 'wp_generate_attachment_metadata' ) ) { require_once( ABSPATH . 'wp-admin/includes/image.php' ); } $r = false; // By default, wp does not scale up if the original file is smaller than the requested thumbnail image size // regardless of the add_image_size setting if ($enforce_enlarge) { add_filter( 'image_resize_dimensions', 'alx_thumbnail_upscale', 10, 6 ); } // regenerate all image sizes and metadata $regenerated_metadata = wp_generate_attachment_metadata( $attachment_id, $fullsizepath ); // merge original metadata $regenerated_metadata['sizes'] = array_merge( $metadata['sizes'], $regenerated_metadata['sizes'] ); //echo 'original metadata'; print_r($metadata); //echo 'regenerated metadata'; print_r($regenerated_metadata); // update metadata if ( wp_update_attachment_metadata( $attachment_id, $regenerated_metadata ) ) { $r = true; //print 'regenerate metadata result successful'; } else { //print 'regenerate metadata result not successful'; } // remove filter so that wp is back to default for thumbnail resizing if ($enforce_enlarge) { remove_filter( 'image_resize_dimensions', 'alx_thumbnail_upscale', 10, 6 ); } return $r; } else { //printf('Image size "%s" exists, regeneration is not needed', $check_image_size); // print_r($metadata['sizes'][$check_image_size]); } }
3.16.18.2. wp_get_image_editor( string $path, array $args = array() ) : WP_Image_Editor|WP_Error
Returns WP_Error or WP_Image_Editor
function image_resize( $file, $max_w, $max_h, $crop = false, $suffix = null, $dest_path = null, $jpeg_quality = 90 ) { _deprecated_function( __FUNCTION__, '3.5.0', 'wp_get_image_editor()' ); $editor = wp_get_image_editor( $file ); if ( is_wp_error( $editor ) ) return $editor; $editor->set_quality( $jpeg_quality ); $resized = $editor->resize( $max_w, $max_h, $crop ); if ( is_wp_error( $resized ) ) return $resized; $dest_file = $editor->generate_filename( $suffix, $dest_path ); $saved = $editor->save( $dest_file ); if ( is_wp_error( $saved ) ) return $saved; return $dest_file; }
3.16.19. HTTP Request API wp-includes/httpl.php wp:api:http request
3.16.19.1. wp_remote_get, wp_remote_post, wp_remote_head, wp_remote_request
$response = wp_remote_get( 'http://www.example.com/index.html' ); // url has to include protocal e.g. http:// if ( is_array( $response ) ) { $header = $response['headers']; // array of http header lines $body = $response['body']; // use the content } wp_remote_get( 'http://www.example.com/index.php?action=foo', array( 'timeout' => 120, 'httpversion' => '1.1' ) );
Methods to work with $response
- wp_remote_retrieve_body() - Retrieves just the body from the response.
- wp_remote_retrieve_header() - Gives you a single HTTP header based on name from the response.
- wp_remote_retrieve_headers() - Returns all of the HTTP headers in an array for processing.
- wp_remote_retrieve_response_code() - Gives you the number for the HTTP response. This should be 200, but could be 4xx or even 3xx on failure.
- wp_remote_retrieve_response_message() - Returns the response message based on the response code.
Default options
global $wp_version; $args = array( 'timeout' => 5, // seconds 'redirection' => 5, // how many times 'httpversion' => '1.0', 'user-agent' => 'WordPress/' . $wp_version . '; ' . home_url(), 'blocking' => true, // php will stop until the request is processed. false to just ping but wp_remote_get will return false 'headers' => array(), 'cookies' => array(), 'body' => null, 'compress' => false, 'decompress' => true, 'sslverify' => true, 'stream' => false, 'filename' => null );
3.16.20. Widget API wp-includes/widgets.php wp:api:widgets
3.16.20.1. dynamic_sidebar vs get_sidebar wp:f:dynamic_sidebar
- Refer to wp:f:get_sidebar
get_sidebarlooks for a template file which should havedynamic_sidebar('sidebar-slug')
3.16.20.2. register_sidebar wp:f:register_sidebar
Create a sidebar (widget area) where widget instances can be assigned to in wp admin UI.
- Display sidebar
add_action( 'widgets_init', 'theme_slug_widgets_init' ); function theme_slug_widgets_init() { register_sidebar( array( 'name' => __( 'Main Sidebar', 'theme-slug' ), 'id' => 'sidebar-1', 'description' => __( 'Widgets in this area will be shown on all posts and pages.', 'theme-slug' ), 'class' => 'tal', // prepend so CSS result for WP Admin page becomes 'sidebar-tal', (default: empty) 'before_widget' => '<li id="%1$s" class="widget %2$s">', 'after_widget' => '</li>', 'before_title' => '<h2 class="widgettitle">', 'after_title' => '</h2>', ) ); }
3.16.20.3. register_widget wp:f:register_widget
- Used in wp:action:widgets_init
- System widgets
wp-includes/widgets/class-wp-widget-*.phpand they're registered atwp-includes/widgets.php
<?php // Register and load the widget function wpb_load_widget() { register_widget( 'wpb_widget' ); } add_action( 'widgets_init', 'wpb_load_widget' ); // Creating the widget class wpb_widget extends WP_Widget { function __construct() { // PHP 5.x //$this->WP_Widget( 'dokan-category-menu', 'Dokan: Product Category', $widget_ops ); parent::__construct( // Base ID of your widget 'wpb_widget', // Widget name will appear in UI __( 'WPBeginner Widget', 'wpb_widget_domain' ), // Widget description [ 'description' => __( 'Sample widget based on WPBeginner Tutorial', 'wpb_widget_domain' ), ] ); } // Creating widget front-end public function widget( $args, $instance ) { $title = apply_filters( 'widget_title', $instance['title'] ); // before and after widget arguments are defined by themes echo $args['before_widget']; if ( ! empty( $title ) ) { echo $args['before_title'] . $title . $args['after_title']; } // This is where you run the code and display the output echo __( 'Hello, World!', 'wpb_widget_domain' ); echo $args['after_widget']; } // Widget Backend public function form( $instance ) { if ( isset( $instance['title'] ) ) { $title = $instance['title']; } else { $title = __( 'New title', 'wpb_widget_domain' ); } // Widget admin form ?> <p> <label for="<?php echo $this->get_field_id( 'title' ); ?>"><?php _e( 'Title:' ); ?></label> <input class="widefat" id="<?php echo $this->get_field_id( 'title' ); ?>" name="<?php echo $this->get_field_name( 'title' ); ?>" type="text" value="<?php echo esc_attr( $title ); ?>"/> </p> <?php } // Updating widget replacing old instances with new public function update( $new_instance, $old_instance ) { $instance = []; $instance['title'] = ( ! empty( $new_instance['title'] ) ) ? strip_tags( $new_instance['title'] ) : ''; return $instance; } } // Class wpb_widget ends here class MyNewWidget extends WP_Widget { function __construct() { // Instantiate the parent object } function widget( $args, $instance ) { // Widget output } function update( $new_instance, $old_instance ) { // Save widget options } function form( $instance ) { // Output admin widget options form } } function myplugin_register_widgets() { register_widget( 'MyNewWidget' ); } add_action( 'widgets_init', 'myplugin_register_widgets' );
3.16.21. Navigation Menu Functions wp-includes/nav-menu.php,nav-menu-template.php
3.16.21.1. register_nav_menus, register_nav_menu, wp_nav_menu
wp:f:register_nav_menus used in wp:action:init or wp:action:after_setup_theme Register custom nav menus This is to create a theme location or navigation menu. Each theme location can have one menu created from wp-admin. Menu created from wp-admin can associate with multiple theme locations. The following registers 3 theme locations. Other translation plugins like Polylang can create theme locations for each extra language based on the already defined theme locations.
function register_my_menus() { register_nav_menus( array( 'main-menu' => __('Main Menu'), 'about-menu' => __('About Menu'), 'services-menu' => __('Services Menu') ) ); }
Display in template
- https://developer.wordpress.org/reference/functions/wp_nav_menu/
- wp:filter:wp_nav_menu_args
- container
- 'div' use what to wrap the menu, bool can be used
- menu_class
- 'menu', css class, e.g. 'class-1 class-2'
- fallback_cb
- false call a cb function if menu doesn't exist e.g. 'WP_Bootstrap_Navwalker::fallback'
- walker
- object e.g.
new WP_Bootstrap_Navwalker()
- wp:filter:wp_nav_menu_container_allowedtags
- wp:filter:wp_nav_menu_objects
- wp:filter:wp_nav_menu_items
- wp:filter:wp_nav_menu_$menu->slug_items
// already echo wp_nav_menu( array( 'theme_location' => 'header-menu' ) );
3.16.22. Toolbar API wp-includes/admin-bar.php wp:api:toolbar
3.16.22.1. show_admin_bar wp:f:show_admin_bar
Use it in functions.php or wp:filter:show_admin_bar
if ( ! current_user_can( 'manage_options' ) ) { show_admin_bar( false ); }
3.16.23. Pluggable functions wp-includes/pluggable.php wp:pluggable
- These functions can be defined in plugins. If all active plugins don't define any of these functions, WP will define them
3.16.23.1. wp_get_current_user() wp:f:wp_get_current_user
Set wp:global:current_user if it's not set. If not logged in, set to 0
$current_user = wp_get_current_user(); if ( 0 == $current_user->ID ) { // Not logged in. } else { // Logged in. }
- Also return wp:user
3.16.23.2. wp_mail() wp:f:wp_mail
Syntax
wp_mail( $to, $subject, $message, $headers = '', $attachments = array() )
- Used filters
wp:filter:wp_mail_from
- From Email address is first defined in
$headersif it hasFrom: your-email@a.ca
user@a.ca,Firstname Lastname <user@a.ca>
- From Email address is first defined in
3.16.24. Administration API wp-admin/includes/admin.php wp:api:core admin
- Administration Plugin API
wp-admin/includes/plugin.phpwp:api:admin:plugin
3.16.24.1. add_meta_box( $id, $title, $callback, $screen = null, $context = 'advanced', $priority = 'default', $callback_args = null ) wp:f:add_meta_box
Used in wp:action:add_meta_boxes
- context
- advanced
- default
- side
- normal
- Priority
- default
- default
- high
- right after title
- (no term)
- low
/* * @param string $id Meta box ID (used in the 'id' attribute for the meta box). * @param string $title Title of the meta box. * @param callable $callback Function that fills the box with the desired content. * The function should echo its output. * @param string|array|WP_Screen $screen Optional. The screen or screens on which to show the box * (such as a post type, 'link', or 'comment'). Accepts a single * screen ID, WP_Screen object, or array of screen IDs. Default * is the current screen. If you have used add_menu_page() or * add_submenu_page() to create a new screen (and hence screen_id), * make sure your menu slug conforms to the limits of sanitize_key() * otherwise the 'screen' menu may not correctly render on your page. * @param string $context Optional. The context within the screen where the boxes * should display. Available contexts vary from screen to * screen. Post edit screen contexts include 'normal', 'side', * and 'advanced'. Comments screen contexts include 'normal' * and 'side'. Menus meta boxes (accordion sections) all use * the 'side' context. Global default is 'advanced'. * @param string $priority Optional. The priority within the context where the boxes * should show ('high', 'low'). Default 'default'. * @param array $callback_args Optional. Data that should be set as the $args property * of the box array (which is the second parameter passed * * to your callback). Default null. */ function add_meta_box( $id, $title, $callback, $screen = null, $context = 'advanced', $priority = 'default', $callback_args = null ) { }
- context
3.16.24.2. remove_meta_box( $id, $page, $context) wp:f:remove_meta_box
- Return
- string
- authordiv
- categorydiv
- comemntstatusdiv
- commentsdiv
- formatdiv
- pageparentdiv
- postcustom
- postexcerpt
- postimagediv
- revisionsdiv
- slugdiv
- submitdiv
- tagsdiv-post_tag
- tagsdiv-{$tax-name}
- {$tax-name}div
- trackbacksdiv
- …
- string. Type of screen
- post
- page
- attachment
- link
- dashboard
- any custom post type e.g. 'my-product'
- string. 'normal', 'advanced', or 'side'
- Post edit screen
- normal, side and advanced
- Comments screen
- normal and side
- Menu meta boxes
- side
- Return nothing
3.16.24.3. add_management_page( $page_title, $menu_title, $capability, $menu_slug, $function = '' )
- Add sub menu to Tools admin menu
Used in wp:action:admin_menu
add_management_page( 'Page title', // string 'Menu title', // string 'manage_options', // string. Capability that is required. 'menu_slug', // string 'callbackFunct', // string or array($this, 'options_page') );
3.17. Default filters and actions wp:default filters
wp-includes/default-filters.php- wp:action:plugins_loaded
3.18. Filter
3.18.1. post_limits wp:filter:post_limits
For feed, use pre_option_posts_per_rss wp:filter:pre_option
function lili_f_post_limits($limit, $query) { return 'LIMIT 0, 25'; return $limit; }
3.18.2. get_meta_sql wp:filter:get_meta_sql
- Modify meta query sql
- This filter is applied also for WP Tax Query with no
$context. WP_Query constructs WP Tax Query first and then WP_Meta_Query
add_filter( 'get_meta_sql', function($clause, $queries, $type, $primary_table, $primary_id_column, $context) { if (lili_request_cache('trigger_get_meta_sql') === 'meta-query-archive-speaker-page') { // if the query is the query needed to be modified // echo "<pre>"; print_r($clause); echo "</pre>"; // echo "<pre>"; print_r($context); echo "</pre>"; $clause['where'] = ''; } return $clause; } ); // In template // sort on custom fields which might not have values or exist $args = array( 'post_type' => 'speaker', 'posts_per_page' => - 1, 'meta_query' => [ 'speaker_last_name_clause' => [ 'key' => 'speaker_last_name', 'compare' => 'NOT EXISTS', ], 'custom_order_clause' => [ 'key' => 'custom_order', 'type' => 'NUMERIC', 'compare' => 'NOT EXISTS', ], ], 'orderby' => [ 'custom_order_clause' => 'DESC', 'speaker_last_name_clause' => 'ASC', ], ); // use the trigger lili_request_cache('trigger_get_meta_sql', 'meta-query-archive-speaker-page'); $speakers = new WP_Query($args); // remove the trigger lili_request_cache('trigger_get_meta_sql', '');
Sample
[
'join' => "LEFT JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) LEFT JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id AND mt1.meta_key = 'speaker_last_name' ) LEFT JOIN wp_postmeta AS mt2 ON ( wp_posts.ID = mt2.post_id ) LEFT JOIN wp_postmeta AS mt3 ON (wp_posts.ID = mt3.post_id AND mt3.meta_key = 'custom_order' )",
'where' => " AND (
(
wp_postmeta.meta_key = 'speaker_last_name'
OR
mt1.post_id IS NULL
)
AND
(
mt2.meta_key= 'custom_order'
OR
mt3.post_id IS NULL
)
)"
]
3.18.3. posts_clauses wp:filter:posts_clauses
- Modify WP_Query object before it runs
$clauses = (array) apply_filters_ref_array( 'posts_clauses', array( compact( $pieces ), &$this ) );
function lili_posts_clauses($clauses, $query) { /* $clauses => [ 'where' => '...', 'groupby' => '', 'join' => '', 'orderby' => 'wp_posts.post_date DESC', 'distinct' => '', 'fields' => 'wp_posts.*', 'limits' => 'LIMIT 0, 20', ] $query is WP_Query object */ // Identify a certain WP_Query $_s = lili_request_cache('trigger_posts_clauses'); if ($_s == 'situationA') { $_old = $clauses['where']; $clauses['orderby'] = ''; $_limit = explode(",",$clauses['limits']); $clauses['limits']=''; $_limit_feature = end($_limit) - 3; reset($_limit); $_features_and_news = <<<SQL SELECT * FROM (SELECT p2.ID FROM wp_posts p2 WHERE p2.post_type IN ('en-vedette') ORDER BY p2.post_date DESC LIMIT {$_limit_feature} ) AS sq1 UNION SELECT * FROM (SELECT p2.ID FROM wp_posts p2 INNER JOIN wp_term_relationships tr ON tr.object_id = p2.ID INNER JOIN wp_terms t ON t.term_id = tr.term_taxonomy_id WHERE p2.post_type = 'nouvelles' AND t.slug = 'homepage-3-squares' ORDER BY p2.post_date DESC LIMIT 3) AS sq2 SQL; $clauses['where'] = $_old." AND wp_posts.ID IN ($_features_and_news)"; // reset cache lili_request_cache('trigger_posts_clauses', ''); } return $clauses; }
3.18.4. query_vars wp:filter:query_vars
- https://codex.wordpress.org/WordPress_Query_Vars
- Used in wp:f:add_rewrite_tag
- return query variable name in array not values
- class-wp.php
$public_query_vars$private_query_vars mysite.ca/?p=123wherepis a public query var where it can be used in URL and has filter effect while private query var can only be used in PHP:$query = new WP_Query(array( 'post__in' => array(3, 7) // post__in is private ));
3.18.5. request wp:filter:request
- filter the array of parsed query variables
apply_filters( 'request', array $query_vars )- empty array
[ 'post_type' => 'mycpt' ][ 'page' => '', 'mycpt' => 'mypost-slug', 'post_type' => 'mycpt', 'name' => 'mypost-slug' ]
3.18.6. page_link, post_link wp:filter:page_link wp:filter:post_link
The filter is used in get_permalink
// When set to true, a structural link will be returned, rather than the actual URI. Example: http://www.example.com/%postname% // instead of http://www.example.com/my-post function append_query_string( $url, $post, $leavename=false ) { if ( $post->post_type == 'post' ) { $url = add_query_arg( 'foo', 'bar', $url ); } return $url; } add_filter( 'post_link', 'append_query_string', 10, 3 );
// add custom query var function myplugin_register_query_vars( $vars ) { $vars[] = 'key1'; $vars[] = 'key2'; return $vars; } add_filter( 'query_vars', 'myplugin_register_query_vars' ); // use custom query var later function myplugin_pre_get_posts( $query ) { // check if the user is requesting an admin page // or current query is not the main query if ( is_admin() || ! $query->is_main_query() ){ return; } $meta_query = array(); // add meta_query elements if( !empty( get_query_var( 'key1' ) ) ){ $meta_query[] = array( 'key' => 'key1', 'value' => get_query_var( 'key1' ), 'compare' => 'LIKE' ); } if( !empty( get_query_var( 'key2' ) ) ){ $meta_query[] = array( 'key' => 'key2', 'value' => get_query_var( 'key2' ), 'compare' => 'LIKE' ); } if( count( $meta_query ) > 1 ){ $meta_query['relation'] = 'AND'; } if( count( $meta_query ) > 0 ){ $query->set( 'meta_query', $meta_query ); } } add_action( 'pre_get_posts', 'myplugin_pre_get_posts', 1 );
3.18.7. url_to_postid
url_to_postid can take full url. But it may not work for custom post type even though get_permalink and rewrite works.
When URL has url parameter p or other special parameters, it will directly return the post id.
echo url_to_postid(get_permalink()); // the current post id.
function lili_get_post_id($custom_permalink) {
global $wpdb;
$post_id = $wpdb->get_col($wpdb->prepare("SELECT post_id FROM $wpdb->postmeta WHERE meta_key= 'custom_permalink' AND meta_value = '%s';", $custom_permalink ));
return $post_id[0];
}
function lili_url_to_postid($url) {
$url_parse = wp_parse_url($url);
if ($url_parse !== false && !empty($url_parse['path'])) {
$post_id = lili_get_post_id(ltrim(strtolower($url_parse['path']), '/'));
if ($post_id) {
$post_id = (int) $post_id;
return '?p='.$post_id;
}
}
return $url;
}
add_filter( 'url_to_postid', 'lili_url_to_postid', 10, 1);
3.18.8. pre_option_(option_name) wp:filter:pre_option_*
Temporarily alter an option without changing in database.
3.18.9. found_posts wp:filter:found_posts
Modify number of found posts
- Related to wp:theme:pagination
- When
offsetis specified in WP_Query, by defaultfound_postswill not take it into account.
add_filter( 'found_posts', 'myprefix_adjust_offset_pagination', 1, 2 ); function myprefix_adjust_offset_pagination($found_posts, $query) { //Define our offset again... $offset = 10; //Ensure we're modifying the right query object... if ( $query->is_home() ) { //Reduce WordPress's found_posts count by the offset... return $found_posts - $offset; } return $found_posts; }
3.18.10. template_include wp:filter:template_include
function pubx_cuw_template_include($template) { global $wp; try { $_url = explode('/',$wp->request); if (!empty($_url) && count($_url) == 3) { $adunit=$_url[1]; $adsize=$_url[2]; $dfp = array( 'em_CUW_BB_1' => ['95740733/'.$adunit, '300x250'], 'em_CUW_BB_2' => ['95740733/'.$adunit, '300x250'], 'em_CUW_BB_3' => ['95740733/'.$adunit, '300x250'], 'em_CUW_LB_1' => ['95740733/'.$adunit, $adsize], // 640x90, 300x90 'em_CUW_LB_2' => ['95740733/'.$adunit, $adsize], // 640x90, 300x90 ); if (isset($dfp[$adunit]) && strlen($adsize) < 10) { $_ad_unit = $dfp[$adunit][0]; $_ad_size = $dfp[$adunit][1]; $htmlFile = <<<HTML <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Canadian Underwriter Partnership</title> <script src="//code.jquery.com/jquery-1.10.2.js"></script> </head> <body> <script> var rnd=Math.floor(Math.random() * (999999 - 10 +1)) + 10; var theUrl="//pubads.g.doubleclick.net/gampad/adx?iu=/{$_ad_unit}&sz={$_ad_size}&c=" + rnd; var xmlHttp = null; xmlHttp = new XMLHttpRequest(); xmlHttp.open( "GET", theUrl ); xmlHttp.send(); xmlHttp.onreadystatechange = function() { if (xmlHttp.readyState==4 && xmlHttp.status==200) { var xmlDoc = $.parseHTML( xmlHttp.response ); var adlink = $(xmlDoc).find('a').attr('href'); if (typeof adlink == "undefined") adlink = "//www.example.com"; window.location = adlink; } } </script> </body> </html> HTML; global $wp_query; $wp_query->is_404 = false; status_header(200); echo $htmlFile; $new_template = locate_template( ['empty_page.php'] ); return $new_template; } } elseif (other_situation) { // you can just flush a buffer or output something // then set some headers // and a response code // then exit(); } } catch (Exception $e) { return $template; } return $template; } add_filter('template_include', 'pubx_cuw_template_include',99);
3.18.11. flush_rewrite_rules_hard wp:filter:flush_rewrite_rules_hard
Stop wp to modify .htaccess inside the Wordpress block
add_filter( 'flush_rewrite_rules_hard', '__return_false' ); // __return_false is a wp function // remove the filter remove_filter( 'flush_rewrite_rules_hard', 'filter_flush_rewrite_rules_hard', 10, 1 );
3.18.12. tiny_mce_before_init wp:filter:tiny_mce_before_init
Change TinyMCE settings https://codex.wordpress.org/TinyMCE Default setting https://www.tinymce.com/docs/configure/
add_filter( 'tiny_mce_before_init', 'myformatTinyMCE' ); function myformatTinyMCE( $in ) { $in['wordpress_adv_hidden'] = FALSE; // enable advanced edit mode //region elements that are not included will be removed. Add all elements. $opts = '*[*]'; $in['valid_elements'] = $opts; $in['extended_valid_elements'] = $opts; //endregion //region To ensure p tag is not removed when switching between Visual and Text modes $in['wpautop'] = false; $in['indent'] = true; //endregion //region p tag will not be added for any new line. // To create a paragraph in Visual mode, Shift+Enter. Otherwise, enter will simply create a <br> // This helps TinyMCE wraps shortcode on any line with p tag. $in['force_p_newlines'] = false; $in['forced_root_block'] = ''; //endregion return $in; }
3.18.13. the_title wp:filter:the_title
- Used in wp:f:get_the_title
- Built-in filters
- wp:f:wptexturize
- replace some text to HTML entities but skip the replacement in some HTML tags
pre, code, kdb, style, script, tt - wp:f:convert_chars
- deal with
&to ensure HTML entities are correct - (no term)
- trim
add_filter( 'the_title', function( $title, $id ) { // $id The post ID return $title; },10,2);
3.18.14. the_title_rss wp:filter:the_title_rss
- Used in wp:f:the_title_rss
add_filter( 'the_title_rss', function( $title ) { return $title; },10);
3.18.15. the_content wp:filter:the_content
Refer to
- wp:f:shortcode_unautop
- wp:f:wpautop
- wp:filter:tiny_mce_before_init
- Change the content of the post after retrieved from db and before it's printed to the screen
- Doc
add_filter( 'the_content', 'wpautop'); // remove_filter( 'the_content', 'wpautop'); add_filter( 'the_content', 'shortcode_unautop' );
//Insert ads after second paragraph of single post content. add_filter( 'the_content', 'prefix_insert_post_ads' ); function prefix_insert_post_ads( $content ) { $ad_code = '<div>Ads code goes here</div>'; if ( is_single() && ! is_admin() ) { return prefix_insert_after_paragraph( $ad_code, 2, $content ); } return $content; } // Parent Function that makes the magic happen function prefix_insert_after_paragraph( $insertion, $paragraph_id, $content ) { $closing_p = '</p>'; $paragraphs = explode( $closing_p, $content ); foreach ($paragraphs as $index => $paragraph) { if ( trim( $paragraph ) ) { $paragraphs[$index] .= $closing_p; } if ( $paragraph_id == $index + 1 ) { $paragraphs[$index] .= $insertion; } } return implode( '', $paragraphs ); }
3.18.16. upload_mimes wp:filter:upload_mimes
Add mime type so that wp can handle its content-type header to browser for upload and sending.
function cc_mime_types($mimes) {
$mimes['svg'] = 'image/svg+xml';
return $mimes;
}
add_filter('upload_mimes', 'cc_mime_types');
3.18.17. widget_text wp:filter:widget_text
Display wp:f:add_shortcode in widget. Filter content output
function exam_plug_text_replace($content) { $new_content = ''; $new_content.= $content . ':)'; return $new_content; } add_filter('widget_text', 'exam_plug_text_replace');
3.18.18. sidebars_widgets wp:filter:sidebars_widgets
Filters the list of sidebars and their widgets. Changing this will change the widget registeration on db.
function disable_all_widgets($sidebars_widgets) {
//var_dump($sidebars_widgets);
// disable all widget areas
// $sidebars_widgets = array(false);
$sidebars_widgets['sidebar-id-1'] = ['widget_name-1'];
return $sidebars_widgets;
}
add_filter('sidebars_widgets', 'disable_all_widgets');
In order to have an instance of a widget, the easiest way is to create another sidebar with the widget in wp-admin
3.18.19. show_admin_bar wp:filter:show_admin_bar
add_filter('show_admin_bar', '__return_false');
3.18.20. excerpt_more wp:filter:excerpt_more
- Default is
echo '[…]'
function twentyseventeen_excerpt_more( $link ) { if ( is_admin() ) { return $link; } $link = sprintf( '<p class="link-more"><a href="%1$s" class="more-link">%2$s</a></p>', esc_url( get_permalink( get_the_ID() ) ), /* translators: %s: Name of current post */ sprintf( __( 'Continue reading<span class="screen-reader-text"> "%s"</span>', 'twentyseventeen' ), get_the_title( get_the_ID() ) ) ); return ' … ' . $link; } add_filter( 'excerpt_more', 'twentyseventeen_excerpt_more' );
3.18.21. excerpt_length wp:filter:excerpt_length
- 55
- Default number of words
3.18.22. wp_calculate_image_srcset wp:filter:wp_calculate_image_srcset
- Refer to html:img:srcset
$sources = apply_filters( 'wp_calculate_image_srcset', $sources, $size_array, $image_src, $image_meta, $attachment_id ); // sample $sources = [ '300' => [ 'url' => 'https://a.ca/wp-content/uploads/2019/03/image-name-100x100.jpg', 'descriptor' => 'w', 'value' => '300' ], '930' => [ 'url' => 'https://a.ca/wp-content/uploads/2015/12/Harbour-House-Sign.jpg', 'descriptor' => 'w', 'value' => '930' ], // ... ];
3.18.23. wp_calculate_image_srcset_meta wp:filter:wp_calculate_image_srcset_meta
add_filter( 'wp_calculate_image_srcset_meta', function($image_meta, $size_array, $image_src, $attachment_id ) { //var_dump($image_meta); if (isset($image_meta['image_meta']) && isset($image_meta['image_meta']['cpt_last_cropping_data']) && isset($image_meta['image_meta']['cpt_last_cropping_data']['date'])) { foreach ( $image_meta[ 'sizes' ] as $k => $v ) { //var_dump($v); if (isset($v['file']) && $v['file']) { $image_meta[ 'sizes' ][$k]['file'] .= '?lastcrop='.$image_meta['image_meta']['cpt_last_cropping_data']['date']; } } } return $image_meta; }, 11, 4);
3.18.24. wp_calculate_image_sizes wp:filter:wp_calculate_image_sizes
- Filter
sizesattribute value for an image. Refer to html:img:sizes
// wp_calculate_image_sizes( array|string $size, string $image_src = null, array $image_meta = null, int $attachment_id ) function twentyseventeen_content_image_sizes_attr( $sizes, $size ) { $width = $size[0]; if ( 740 <= $width ) { $sizes = '(max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px'; } if ( is_active_sidebar( 'sidebar-1' ) || is_archive() || is_search() || is_home() || is_page() ) { if ( ! ( is_page() && 'one-column' === get_theme_mod( 'page_options' ) ) && 767 <= $width ) { $sizes = '(max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px'; } } return $sizes; } add_filter( 'wp_calculate_image_sizes', 'twentyseventeen_content_image_sizes_attr', 10, 2 );
3.18.25. wp_resource_hints
apply_filters( 'wp_resource_hints', array $urls, string $relation_type )
relation_type :: dns-prefetch, preconnect, prefetch, and prerender
https://make.wordpress.org/core/2016/07/06/resource-hints-in-4-6/
function twentyseventeen_resource_hints( $urls, $relation_type ) {
if ( wp_style_is( 'twentyseventeen-fonts', 'queue' ) && 'preconnect' === $relation_type ) {
$urls[] = array(
'href' => 'https://fonts.gstatic.com',
'crossorigin',
);
}
return $urls;
}
add_filter( 'wp_resource_hints', 'twentyseventeen_resource_hints', 10, 2 );
3.18.26. script_loader_src style_loader_src wp:filter:script_loader_src wp:filter:style_loader_src
Remove current site url from script and style files
add_filter( 'script_loader_src', 'wpse47206_src' );
add_filter( 'style_loader_src', 'wpse47206_src' );
function wpse47206_src( $url ) {
if( is_admin() ) return $url;
return str_replace( site_url(), '', $url );
}
3.18.27. imgae_size_names_choose wp:filter:image_size_names_choose
- Add an wp:f:add_image_size to WYSIWYG image style dropdown list
add_image_size( 'video-thumbnail', 320, 180, true ); add_filter( 'image_size_names_choose', function ( $sizes ) { return array_merge( $sizes, array( 'video-thumbnail' => __( 'Video Thumbnail' ), ) ); });
3.18.28. get_header_image_tag
apply_filters( 'get_header_image_tag', string $html, object $header, array $attr ); function twentyseventeen_header_image_tag( $html, $header, $attr ) { if ( isset( $attr['sizes'] ) ) { $html = str_replace( $attr['sizes'], '100vw', $html ); } return $html; } add_filter( 'get_header_image_tag', 'twentyseventeen_header_image_tag', 10, 3 );
In template
<img src="<?php header_image(); ?>" height="<?php echo get_custom_header()->height; ?>" width="<?php echo get_custom_header()->width; ?>" alt="" />
3.18.29. wp_nav_menu_args wp:filter:wp_nav_menu_args
// $args = apply_filters( 'wp_nav_menu_args', $args ); add_filter( 'wp_nav_menu_args', 'bfg_nav_menu_args_filter' ); function bfg_nav_menu_args_filter( $args ) { require_once( BFG_THEME_MODULES . 'class-wp-bootstrap-navwalker.php' ); if ( 'primary' === $args['theme_location'] ) { $args['container'] = false; $args['menu_class'] = 'navbar-nav mr-auto'; $args['fallback_cb'] = 'WP_Bootstrap_Navwalker::fallback'; $args['walker'] = new WP_Bootstrap_Navwalker(); } return $args; }
3.18.30. wp_get_attachment_image_src wp:filter:wp_get_attachment_image_src
/** * Add URL Parameter lastcrop with current datetime to bust image cache * * Add cache busting if image has metadata cpt_last_cropping_data which is added in filter gncm_crop_thumbnails_before_update_metadata * * @param $image * @param $attachment_id * @param $size * @param $icon * * @return array */ function gncm_wp_get_attachment_image_src( $image, $attachment_id, $size, $icon ) { if ( is_array( $image ) && ! empty( $image ) && ! is_admin() && function_exists( 'http_build_url' ) ) { $url = @parse_url( $image[0] ); $meta = wp_get_attachment_metadata( $attachment_id ); if ( ! empty( $url ) && ! empty( $meta ) && isset( $meta['image_meta'] ) && isset( $meta['image_meta']['cpt_last_cropping_data'] ) && isset( $meta['image_meta']['cpt_last_cropping_data']['date'] ) ) { $url['query'] = ( isset( $url['query'] ) ) ? $url['query'] : ''; parse_str( $url['query'], $params ); $params = array_merge( [ 'lastcrop' => $meta['image_meta']['cpt_last_cropping_data']['date'] ], $params ); $url['query'] = http_build_query( $params ); $image[0] = http_build_url( $url ); } } return $image; } add_filter( 'wp_get_attachment_image_src', 'gncm_wp_get_attachment_image_src', 11, 4 );
3.18.31. wp_get_attachment_image_attributes
apply_filters( 'wp_get_attachment_image_attributes', array $attr, WP_Post $attachment, string|array $size )- (array) Attributes for the image markup.
- (WP_Post) Image attachment post.
- (string|array) Requested size. Image size or array of width and height values (in that order). Default 'thumbnail'.
function twentyseventeen_post_thumbnail_sizes_attr( $attr, $attachment, $size ) { if ( is_archive() || is_search() || is_home() ) { $attr['sizes'] = '(max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px'; } else { $attr['sizes'] = '100vw'; } return $attr; } add_filter( 'wp_get_attachment_image_attributes', 'twentyseventeen_post_thumbnail_sizes_attr', 10, 3 );
3.18.32. widget_tag_cloud_args
function twentyseventeen_widget_tag_cloud_args( $args ) {
$args['largest'] = 1;
$args['smallest'] = 1;
$args['unit'] = 'em';
$args['format'] = 'list';
return $args;
}
add_filter( 'widget_tag_cloud_args', 'twentyseventeen_widget_tag_cloud_args' );
3.18.33. comments_open
3.18.34. deprecated_constructor_trigger_error wp:filter:deprecated_constructor_trigger_error
add_filter('deprecated_constructor_trigger_error', '__return_false');
3.19. Action
3.19.1. activated_plugin wp:action:activated_plugin
- Change plugins loading order. By default is alpha based on plugin's path + plugin's php file
- It reads and updates wp:db:wp_options:active_plugins
- Use the following code at the beginning of a plugin to change the order
function this_plugin_last() { $wp_path_to_this_file = preg_replace('/(.*)plugins\/(.*)$/', WP_PLUGIN_DIR."/$2", __FILE__); $this_plugin = plugin_basename(trim($wp_path_to_this_file)); $active_plugins = get_option('active_plugins'); $this_plugin_key = array_search($this_plugin, $active_plugins); array_splice($active_plugins, $this_plugin_key, 1); array_push($active_plugins, $this_plugin); update_option('active_plugins', $active_plugins); } add_action("activated_plugin", "this_plugin_last");
3.19.2. plugins_loaded wp:action:plugins_loaded
- default added actions
wp_maybe_load_widgetswp_maybe_load_embeds_wp_customize_include
3.19.3. after_setup_theme wp:action:after_setup_theme
Called each page load after the theme is initialized. Do basic setup, registration and init actions for a theme
May run these in this action
3.19.4. init wp:action:init
- Fires after Wordpress has finished loading but before any headers are sent
- User is already authenticated
- Useful for intercepting
$_GETor$_POST - Functions or actions
3.19.5. widgets_init wp:action:widgets_init
Refer to wp:filter:sidebars_widgets to modify registered sidebars and their widgets. Refer to wp:f:register_widget to create a widget
function lili_widgets_init() { $args = array( 'name' => __( 'Sidebar name', 'theme_text_domain' ), 'id' => 'unique-sidebar-id', // %1$s 'description' => '', 'class' => '', // %2$s, will have ~sidebar-~ prepended 'before_widget' => '<li id="%1$s" class="widget %2$s">', 'after_widget' => '</li>', 'before_title' => '<h2 class="widgettitle">', 'after_title' => '</h2>' ); register_sidebar( $args ); // register_sidebar( $args2 ); // Create multiple sidebars with the same config $args = array( 'name' => __('Sidebar %d'), 'id' => 'sidebar', 'description' => '', 'class' => '', 'before_widget' => '<li id="%1$s" class="widget %2$s">', 'after_widget' => '</li>', 'before_title' => '<h2 class="widgettitle">', 'after_title' => '</h2>' ); // register_sidebars( 2, $args ); } add_action('widgets_init', 'lili_widgets_init');
Display
<?php if ( is_active_sidebar( 'home_right_1' ) ) : ?> <div id="primary-sidebar" class="primary-sidebar widget-area" role="complementary"> <?php dynamic_sidebar( 'home_right_1' ); ?> </div><!-- #primary-sidebar --> <?php endif; ?>
3.19.6. admin_menu wp:action:admin_menu
3.19.7. admin_init wp:action:admin_init
It's triggered before any other hook when a user accesses the admin area. This hook doesn't provide any parameters, so it can only be used to callback a specified function.
These core functions are called: register_admin_color_schemes send_frame_options_header _wp_check_for_scheduled_split_terms _wp_admin_bar_init _maybe_update_core _maybe_update_plugins _maybe_update_themes
3.19.8. parse_request wp:action:parse_request
3.19.9. pre_get_posts wp:action:pre_get_posts
- Provide a chance to modify
$queryobject (or do something else) before it's run.$querycan be any main (often) or secondary query - Do not use this to alter query for single Page requests
- Add action in functions.php. Query is already made when template is loaded
do_action_ref_array( 'pre_get_posts', array( &$this ) )- Use it instead of
query_posts - Refer to WP_Query
add_action( 'pre_get_posts', 'lili_pre_get_posts' ); // Should not call this in template because in template the $query is already run function lili_pre_get_posts( $query ) { if ( ! is_page() && ! is_home() && $query->is_main_query() ) { $query->set( 'post_type', array( 'post', 'movies' ) ); } // this function should return nothing. }
3.19.10. send_headers wp:action:send_headers
- Refer to pantheon:varnish
$regex_path_patterns = array( '#^/news/?#', '#^/about/?#', ); // Loop through the patterns. foreach ($regex_path_patterns as $regex_path_pattern) { if (preg_match($regex_path_pattern, $_SERVER['REQUEST_URI'])) { add_action( 'send_headers', 'add_header_nocache', 15 ); // No need to continue the loop once there's a match. break; } } function add_header_nocache() { header( 'Cache-Control: no-cache, must-revalidate, max-age=0' ); } /* For WP REST API specific paths, we use a different approach by using the rest_post_dispatch filter */ // wp-json paths or any custom endpoints $regex_json_path_patterns = array( '#^/wp-json/wp/v2?#', '#^/wp-json/?#' ); foreach ($regex_json_path_patterns as $regex_json_path_pattern) { if (preg_match($regex_json_path_pattern, $_SERVER['REQUEST_URI'])) { // re-use the rest_post_dispatch filter in the Pantheon page cache plugin add_filter( 'rest_post_dispatch', 'filter_rest_post_dispatch_send_cache_control', 12, 2 ); // Re-define the send_header value with any custom Cache-Control header function filter_rest_post_dispatch_send_cache_control( $response, $server ) { $server->send_header( 'Cache-Control', 'no-cache, must-revalidate, max-age=0' ); return $response; } break; } }
3.19.11. wp_enqueue_scripts wp:action:wp_enqueue_scripts
Can use wp:f:wp_enqueue_style and wp:f:wp_enqueue_script. Priority might be needed to change to greater 10 to load after default theme/style.css Use wp:filter:script_loader_src and wp:filter:style_loader_src to change the source of scripts and styles
function lili_scripts() { wp_enqueue_style('lili-custom-styles', get_stylesheet_directory_uri().'/css/style.css'); } add_action( 'wp_enqueue_scripts', 'lili_scripts', 20);
3.19.12. admin_head wp:action:admin_head
function my_custom_admin_head() { echo '<style>[for="wp_welcome_panel-hide"] {display: none !important;}</style>'; } add_action( 'admin_head', 'my_custom_admin_head' ); // Add inline JS in the admin head with the <script> tag function my_custom_admin_head() { echo '<script type=""text/javascript">console.log('admin script')</script>'; } add_action( 'admin_head', 'my_custom_admin_head' );
3.19.13. template_redirect wp:action:template_redirect
Refer to wp:php:redirect
3.19.14. wp_head, wp_footer, get_footer wp:action:wp_head
- These actions are good for adding inline CSS and JavaScripts
- Refer to wp:default filters
wp_head();andwp_footer();are usually called in template filesheader/footer.php- default
wp_footeractions wp_print_footer_scriptsp:20 andwp_admin_bar_renderp:1000
- default
get_footeraction is called at the beginning of theget_footer();function, which will includefooter.phpfile later
If a parameter is passed, footer-[$name].php is also included and you can play around $name in the get_footer hook like this
<?php function themeslug_footer_hook( $name ) { if ( 'new' == $name ) { ?> <script> (function($) { //put all your jQuery goodness in here. })(jQuery); </script> <?php } // you can also print if ( is_singular() && pings_open() ) { printf( '<link rel="pingback" href="%s">' . "\n", get_bloginfo( 'pingback_url' ) ); } } add_action( 'get_footer', 'themeslug_footer_hook' );
Remove emoji javascript and style
remove_action('wp_head', 'print_emoji_detection_script', 7); remove_action('wp_print_styles', 'print_emoji_styles'); remove_action('wp_head', 'wlwmanifest_link'); // Windows Live Writer support remove_action('wp_head', 'wp_generator'); // wp version remove_action ('wp_head', 'rsd_link'); // remove EditURI/RSD Really Simple Directory and xmlrpc.php remove_action( 'wp_head', 'wp_shortlink_wp_head'); // e.g. http://yoursite.com/?p=7 // By default, wp:f:add_theme_support automatic-feed-links is not enabled and the feed links are not inserted remove_action( 'wp_head', 'feed_links', 2 ); // e.g. <link rel="alternate" type="application/rss+xml" title="WP Site » Feed" href="http://localhost/wp/feed/" /> remove_action('wp_head', 'feed_links_extra', 3 ); // remove comment feeds e.g. Like: <link rel="alternate" type="application/rss+xml" title="WP Site » Comments Feed" href="http://localhost/wp/comments/feed/" /> remove_action('wp_head', 'adjacent_posts_rel_link'); // remove PREV and NEXT links e.g. <link rel='prev' title='The Post Before This One' href='http://localhost/wp/?p=4' /> remove_action( 'wp_head','rest_output_link_wp_head'); // wp-json
3.19.15. add_meta_boxes, add_meta_boxespost_type wp:action:add_meta_boxes
lili_add_meta_boxes($post_type, $post)takes 2 arguments, no returnlili_add_meta_boxes_{post_type}($post)takes 1 argument, no return- Use
- or modify
global $wp_meta_boxes $wp_meta_boxes['post-type']['normal']['core']['postexcerpt']['title'] = 'Your title';- You can change the title, callback (complete change to another function) and different context.
- The callback most likely outputs a label, form field and some description.
- You can either completely change the callback or use wp:filter:gettext to change the translation
3.19.16. edit_page_form wp:action:edit_page_form
Fires after 'normal' context meta boxes have been output for the 'page' post type. Control admin Edit Page form.
function volunteer_hours_editor() {
$content = get_post_meta(get_the_ID(), 'volunteer-hours', true);
$settings = array('textarea_name' => 'volunteer-hours');
//wpautop($content);
wp_editor( $content, 'volunteer-hours', $settings);
}
3.19.17. edit_form_advanced
wp:action:edit_form_advanced Same as wp:action:edit_page_form but for all post types other than 'page'
3.19.18. save_post wp:action:save_post
- Called when a post or page is created or updated, which could be from an import, post/page edit form, xmlrpc, or post by email
- The data for the post is stored in
$_POST,$_GETorglobal $post_data, depending on how the post was edited. e.g. quick edits use$_POST - Use
add_action( 'save_post', 'save_post', 10, 2 ); function save_post( $post_id, $post ) { if ( !current_user_can( 'edit_post', $post_id ) ) { return $post_id; // end the action. Not necessary to return. } // update a custom field update_post_meta( $post_id, 'volunteer-hours', stripslashes( wpautop($_POST['volunteer-hours']) ) ); // auto select parent term if child term is selected for a post type if ($post->post_type == 'product') { $tax = 'product_category'; $terms = wp_get_post_terms($post_id, $tax); foreach ($terms as $term) { while ($term->parent != 0 && !has_term( $term->parent, $tax, $post )) { // move upward until we get to 0 level terms // if parent is already selected, don't add wp_set_post_terms($post_id, array($term->parent), $tax, true); $term = get_term($term->parent, $tax); } } } // auto select parent term. }
3.19.19. do_feed_$feedname wp:action:do_feed_$feedname
- Refer to wp:f:do_feed and wp:f:add_feed
3.20. FTP setting wp:ftp
WordPress is usually run using Apache or Nginx user which might not have permission to write files. docker container map /host/path/to/html:/var/www Setup these in wp-config.php
define('FTP_BASE', '/host/path/to/html/'); define('FS_METHOD', 'direct'); define('FTP_HOST', '8.8.8.8'); // public ip define('FTP_USER', 'ftpuser'); define('FTP_PASS', 'ftppassword'); define('FTP_SSL', true);
https://codex.wordpress.org/Editing_wp-config.php#WordPress_Upgrade_Constants outputs comment form
3.21. Sanitizing and Escaping User Data
https://codex.wordpress.org/Validating_Sanitizing_and_Escaping_User_Data
Sanitize data before it's saved
<input type="text" id="title" name="title" /> $title = sanitize_text_field( $_POST['title'] ); update_post_meta( $post->ID, 'title', $title ); sanitize_email() sanitize_file_name() sanitize_html_class() sanitize_key() sanitize_meta() sanitize_mime_type() sanitize_option() sanitize_sql_orderby() sanitize_text_field() sanitize_textarea_field() sanitize_title() sanitize_title_for_query() sanitize_title_with_dashes() sanitize_user()
Escape
- Translate then escape
esc_html__( $text, $domain = 'default')- Escape for HTML element attribute value
esc_attr- Escape js for HTML element attribute value
esc_js- (no term)
- Escape for
<textarea>value
<h4><?php echo esc_html( $title ); ?></h4> <img src="<?php echo esc_url( $great_user_picture_url ); ?>" /> <a href="#" onclick="<?php echo esc_js( $custom_js ); ?>">Click me</a> <ul class="<?php echo esc_attr( $stored_class ); ?>"> <textarea><?php echo esc_textarea( $text ); ?></textarea>
3.22. Ajax wp:ajax
3.22.1. Ajax on wp admin pages
Javascript global variable ajaxurl is defined on admin pages
functions.php
add_action( 'admin_enqueue_scripts', 'add_ajax_file'); function add_ajax_file() { wp_enqueue_script( 'ajax_custom_script', plugins_url( '/js/my.js' , __FILE___), array('jquery') ); } add_action( 'wp_ajax_my_action', 'my_action'); function my_action() { global $wpdb; $data = intval($_POST['data']); echo $data; // echo json_encde($data); wp_die(); }
my.js
jQuery(document).ready(function($) { var data = { 'action': 'my_action', 'data': 'some data' } $.post(ajaxurl, data, function(r) { console.log('response:', r); }); })
3.22.2. Ajax on wp front end pages
Define global variable mynamespace.ajaxurl functions.php
add_action( 'wp_enqueue_scripts', 'add_custom_namespace' ); function add_custom_namespace() { wp_localize_script( 'jquery', 'mynamespace', array( 'ajaxurl' => admin_url('admin-ajax.php'), 'data' => 'some data' ) ); }
my.js
jQuery(document).ready(function($) { var data = { 'action': 'my_action', 'data': mynamespace.data } $.post(mynamespace.ajaxurl, data, function(r) { console.log('response:', r); }); })
functions.php
add_action( 'wp_ajax_my_action', 'my_action'); add_action( 'wp_ajax_nopriv_my_action', 'my_action'); // non-logged in users // same my_action
3.23. Theme wp:theme
3.23.1. child theme
https://codex.wordpress.org/Child_Themes wp-content/themes/parent-theme-name-child wp-content/themes/parent-theme-name-child/style.css
For new theme, index.php is required
/* Theme Name: Twenty Fifteen Child Theme URI: http://example.com/twenty-fifteen-child/ Description: Twenty Fifteen Child Theme Author: John Doe Author URI: http://example.com Template: twentyfifteen Version: 1.0.0 License: GNU General Public License v2 or later License URI: http://www.gnu.org/licenses/gpl-2.0.html Tags: light, dark, two-columns, right-sidebar, responsive-layout, accessibility-ready Text Domain: twenty-fifteen-child */
Only Theme Name and Template (the name of the parent theme directory) are required
functions.php in child loads first then the parent's
add_action( 'wp_enqueue_scripts', 'my_theme_enqueue_styles' ); function my_theme_enqueue_styles() { $parent_style = 'parent-style'; // This is 'twentyfifteen-style' for the Twenty Fifteen theme. wp_enqueue_style( $parent_style, get_template_directory_uri() . '/style.css' ); // child theme style.css is loaded after parent's style.css wp_enqueue_style( 'child-style', get_stylesheet_directory_uri() . '/style.css', array( $parent_style ), wp_get_theme()->get('Version') ); }
That requires parent functions.php to have
if ( ! function_exists( 'theme_special_nav' ) ) { function theme_special_nav() { // Do something. } }
3.23.2. functions.php wp:theme:functions
3.23.3. Template Files
- All Template Files
- https://developer.wordpress.org/themes/basics/template-hierarchy/
- Another WordPress Template Hierarchy
single.php single-{post-type}.php archive.php archive-{post-type}.php
if single.php and archive.php don't exist, wp looks for index.php
3.23.4. Page template
- Hierarchy
- Global Custom Page Template
- can be applied to multiple pages. Set in Edit Post > Page Attributes > Template
- page-{slug}.php
- for one specific page. Directly put them in theme folder
- page-{id}.php
- for one specific page. Directly put them in theme folder
- page.php
- refer to conditional tags
- (no term)
- singular.php
- (no term)
- index.php
- Attachment
MIME_type.php
- image.php, video.php, application.php
- For text/plain, in order
- text_plain.php
- plain.php
- text.php
- attachment.php
- single-attachment.php
- single.php
- singular.php
3.23.4.1. Global Custom Page Template wp:custom page template
- Put these global template files in theme subfolder
page-templates(or any .php files) - File name can be anything but follow
page_two-columns.php. Don't usepage-as prefix! - Make a copy of page.php to start
- Without
Template Post Type, global custom page template is only available to post typepage
<?php /** * Template Name: Full Width Page * Template Post Type: post, page, product * * @package WordPress * @subpackage Twenty_Fourteen * @since Twenty Fourteen 1.0 */
3.23.4.2. archive.php
<?php /** * The template for displaying archive pages * * @link https://codex.wordpress.org/Template_Hierarchy * * @package WordPress * @subpackage Twenty_Seventeen * @since 1.0 * @version 1.0 */ get_header(); ?> <div class="wrap"> <?php if ( have_posts() ) : ?> <header class="page-header"> <?php the_archive_title( '<h1 class="page-title">', '</h1>' ); the_archive_description( '<div class="taxonomy-description">', '</div>' ); ?> </header><!-- .page-header --> <?php endif; ?> <div id="primary" class="content-area"> <main id="main" class="site-main" role="main"> <?php if ( have_posts() ) : ?> <?php /* Start the Loop */ while ( have_posts() ) : the_post(); /* * Include the Post-Format-specific template for the content. * If you want to override this in a child theme, then include a file * called content-___.php (where ___ is the Post Format name) and that will be used instead. */ get_template_part( 'template-parts/post/content', get_post_format() ); endwhile; the_posts_pagination( array( 'prev_text' => twentyseventeen_get_svg( array( 'icon' => 'arrow-left' ) ) . '<span class="screen-reader-text">' . __( 'Previous page', 'twentyseventeen' ) . '</span>', 'next_text' => '<span class="screen-reader-text">' . __( 'Next page', 'twentyseventeen' ) . '</span>' . twentyseventeen_get_svg( array( 'icon' => 'arrow-right' ) ), 'before_page_number' => '<span class="meta-nav screen-reader-text">' . __( 'Page', 'twentyseventeen' ) . ' </span>', ) ); else : get_template_part( 'template-parts/post/content', 'none' ); endif; ?> </main><!-- #main --> </div><!-- #primary --> <?php get_sidebar(); ?> </div><!-- .wrap --> <?php get_footer();
3.23.5. Pagination wp:theme:pagination
https://developer.wordpress.org/themes/functionality/pagination/
the_post_pagination( $args = array() )echo get_the_posts_pagination( $args )paginate_links( string|array $args = '' )- old version of
the_post_pagination - get_the_posts_pagination()
- wp:f:get_the_posts_pagination. Calls
paginate_links - posts_nav_link()
next_posts_link()echo get_next_posts_link()previous_posts_link()echo get_previous_posts_link()
<?php if ( have_posts() ) : ?> <!-- Add the pagination functions here. --> <!-- Start of the main loop. --> <?php while ( have_posts() ) : the_post(); ?> <!-- the rest of your theme's main loop --> <?php endwhile; ?> <!-- End of the main loop --> <!-- Add the pagination functions here. --> <div class="nav-previous alignleft"><?php next_posts_link( 'Older posts' ); ?></div> <div class="nav-next alignright"><?php previous_posts_link( 'Newer posts' ); ?></div> <?php else : ?> <?php _e('Sorry, no posts matched your criteria.'); ?> <?php endif; ?>
Custom Query
$paged = (get_query_var('paged')) ? get_query_var('paged') : 1; $args = array( 'paged' => $paged, 'orderby' => 'meta_value_num', 'meta_key' => 'post_views_count', 'posts_per_page' => 36 ); global $wp_query; $loop = new WP_Query($args); if ($loop->have_posts()) : while ($loop->have_posts()) : $loop->the_post(); get_template_part('content', get_post_format()); endwhile; $wp_query = $loop; the_posts_navigation(); endif; wp_reset_query(); // reset main query to original
3.23.5.1. get_the_posts_pagination wp:f:get_the_posts_pagination
get_the_posts_pagination( array $args = array() )
3.23.6. Setup
3.23.7. Include header, footer, sidebar and other, search form
3.23.9. Functions used in template
Refer to wp:template tags and wp:t:general
3.23.10. Previous themes
- Newspaper
- JobRoller
- Doc, all articles
- Post Types
- job_listing
- Tax: job_cat, job_type, job_tag, job_salary
- resume
- Tax: resume_category, resume_job_type, resume_languages, resume_specialities, resume_groups
- pricing-plan
- custom-form
- job_listing
3.24. Genesis Framework wp:genesis
- Code snippets (examples)
- https://my.studiopress.com/documentation/snippets/
- function genesis()
- lib/framework.php
- (no term)
- hooks
- Shortcodes
- https://genesistutorials.com/genesis-shortcode-list/ search for
add_shortcode - Changelog
- https://studiopress.github.io/genesis/changelog/
3.24.1. Functions wp:genesis:functions
lib/functions/*.php
3.24.1.1. upgrade.php
3.24.1.2. compat.php
3.24.1.3. general.php
genesis_a11y( $arg = 'screen-reader-text' )- wp:genesis:theme support:genesis-accessibility, return bool
3.24.1.4. options.php wp:genesis:options
header_scripts- string
footer_scripts- string
content_archive- full or excerpts
content_archive_thumbnail- 0, not display thumbnail
image_size- 'thumbnail', default thumbnail size
breadcrumb_home- 0 or 1
breadcrumb_singlebreadcrumb_archivebreadcrumb_pagebreadcrumb_attachment- (no term)
breadcrumb_404
3.24.1.5. image.php
3.24.1.6. markup.php wp:genesis:functions:markup.php
genesis_markup( $args=[] )
$args- echo
- true
- html5
- echo and run
genesis_attr( $args['context'] )if theme supporthtml5is enabled otherwise xhtml is echoed - xhtml
- context
- open
- close
- content
$pre = apply_filters( "genesis_markup_{$args['context']}", false, $args ); // short circuit- $open
genesis_attr( $args['context'], array(), $args)- filter
genesis_markup_$context_open( $open, $args )
- $content
- filter
genesis_markup_$context_content( $args['content'], $args )
- filter
- $close
- filter
genesis_markup_$context_close( $args['close'], $args )
- filter
- filter
genesis_markup_open( $open, $arg ) - filter
genesis_markup_close( $close, $args ) - echo or return
$open.$content.$close
genesis_attr( $context, $attributes = array(), $args = array() )wp:genesis:functions:genesis_attr
genesis_parse_attr()calculates key/value pairs of attribute and value$contextbecomesclassattributegenesis_attr_$context( $attributes, $context, $args )
genesis_attr_$context_output( $output, $attributes, $context, $args)- filter
genesis_attr_$context()- built in
genesis_attr_site-header
// built in example for site-header function genesis_attributes_header( $attributes ) { $attributes['itemscope'] = true; $attributes['itemtype'] = 'https://schema.org/WPHeader'; return $attributes; }
genesis_attr_site-footergenesis_attr_footer-widgets- add
wrapto HTML attributeclassfor wp:genesis:theme support:genesis-structural-wraps
- Then filter
genesis_structural_wrap-{$context}is called latergenesis_attr_blog-template-descriptiongenesis_attr_entry-meta-before-contentgenesis_attributes_entry()genesis_attr_entry-contentgenesis_attr_entry-imagegenesis_attr_entry-image-linkgenesis_attr_bodygenesis_attr-nav-$theme_locationgenesis_attr-breadcrumb
- filter
genesis_attr_$context_output( $output, $attributes, $context, $args )
- Sample genesis_attr_body: genesis_attributes_body
add_filter( 'genesis_attr_body', 'genesis_attributes_body' ); /** * Add attributes for body element. * * @since 2.0.0 * * @param array $attributes Existing attributes for `body` element. * @return array Amended attributes for `body` element. */ function genesis_attributes_body( $attributes ) { $attributes['class'] = implode( ' ', get_body_class() ); $attributes['itemscope'] = true; $attributes['itemtype'] = 'https://schema.org/WebPage'; // Search results pages. if ( is_search() ) { $attributes['itemtype'] = 'https://schema.org/SearchResultsPage'; } return $attributes; }
3.24.1.7. breadcrumb.php
genesis_do_breadcrumbs()genesis_breadcrumb( $args = [] )
$args- prefix
- (no term)
labels['prefix']genesis_markup- open
divcontext:breadcrumb
- (no term)
$this->build_crumbs()- filter
genesis_build_crumbs $crumbs,$this->args
- filter
- (no term)
suffixgenesis_markup- close
divcontext:breadcrumb
get_archive_crumb()- filter
genesis_archive_crumb
3.24.1.8. menu.php
genesis_nav_menu( $args )echo genesis_get_nav_menu( $args );wp:genesis:functions:genesis_nav_menu- (no term)
echo genesis_get_nav_menu( $args )- add superfish
$nav_markup_open = genesis_structural_wrap( 'menu-' . $sanitized_location, 'open', 0 );wp:genesis:functions:genesis_structural_wrap$nav = wp_nav_menu( $args )$nav_markup_close = genesis_structural_wrap( 'menu-' . $sanitized_location, 'close', 0 )- wrap with
<nav></nav>filtergenesis_attr-nav-$theme_location, content - filter
genesis_do_nav( $nav_output, $nav, $args )orgenesis_do_subnav( $nav_output, $nav, $args )
3.24.1.9. layout.php wp:genesis:functions:layout.php
genesis_structural_wrap( $context = '', $output = 'open', $echo = true )wp:genesis:functions:genesis_structural_wrap- echo or return a structural wrap div
- Check if has wp:genesis:theme support:genesis-structural-wraps
- or
</div>">modify attr usinggenesis_attr( 'structural-wrap' ) Finally, filter output
genesis_structural_wrap-$context( $output, $original_output )// no built in example for filter genesis_structural_wrap-$context // custom code add_filter( 'genesis_structural_wrap-archive-col1', function($output, $original_output) { if ( 'open' == $original_output ) { $output = '<div class="wrap archive-wrapper archive-col1">'; } return $output; }, 10, 2 );
genesis_create_initial_layouts()- content-sidebar (default)
- sidebar-content
- content-sidebar-sidebar
- sidebar-sidebar-content
- sidebar-content-sidebar
genesis_get_sidebar()will not load the sidebargenesis_register_layout
genesis_register_layout( $id = '', $args = array() )genesis_site_layout( $use_cache = true )- filter
genesis_site_layoutto short circuit - get layout setting from post, term,
genesis_get_cpt_option( 'layout' ) - If not set above, get it from
genesis_get_option( 'site_layout')- filter
genesis_pre_get_option_site_layoutadd_filter( 'genesis_pre_get_option_site_layout', '__genesis_return_content_sidebar' );__genesis_return_full_width_content
- filter
- filter
3.24.1.10. formatting.php
3.24.1.11. seo.php
3.24.1.12. widgetize.php
genesis_register_widget_area( $args )orgenesis_register_sidebar- Use wp:f:register_sidebar
- Defined in wp:genesis:action:genesis_setup
- before_widget, after_widget, before_title, after_title
genesis_register_sidebar_defaultsgenesis_register_wdiget_area_defaults- Overide
$defaultsusing$args - default registered widget areas
- header-right
- sidebar
- sidebar-alt
- after-entry
- wp:f:dynamic_sidebar
- return false if sidebar isn't found
- filter
genesis_widget_area_defaults, [ before, after, default, show_active, before_sidebar_hook, after_sidebar_hook] - action
genesis_before_$id_widget_areagenesis_after_$id_widget_area
3.24.1.13. feed.php
3.24.1.14. toolbar.php
3.24.1.15. head.php
3.24.2. Structure
3.24.2.1. header.php
3.24.2.2. footer.php
3.24.2.3. menu.php
genesis_do_nav()wp:genesis:structure:genesis_do_nav- display primary nav menu. Use wp:genesis:functions:genesis_nav_menu
genesis_do_subnav()wp:genesis:structure:genesis_do_subnav
3.24.2.4. layout.php
Depends on genesis_site_layout in wp:genesis:functions:layout.php
- load primary sidebar
- load sidebar-alt
- add layout name as body_class
3.24.2.5. post.php wp:genesis:structure:post.php
genesis_do_post_format_image()
Display post format icon
genesis_post_info()
Display post info (byline)
genesis_do_post_content()
- is_singular()
the_content()- filter
genesis_edit_post_link
'excerpts' === genesis_get_option( 'content_archive' )the_excerpt()
- is_singular()
genesis_reset_loops()
- add actions
- genesis_entry_header genesis_do_post_format_image p:4
- genesis_entry_header genesis_entry_header_markup_open p:5
- genesis_entry_header genesis_entry_header_markup_close p:15
- genesis_entry_header genesis_do_post_title
- genesis_entry_header genesis_post_info p:12
- genesis_entry_content genesis_do_post_image p:8
- genesis_entry_content genesis_do_post_content
- genesis_entry_content genesis_do_post_content_nav p:12
- genesis_entry_content genesis_do_post_permalink p:14
- genesis_entry_footer genesis_entry_footer_markup_open p:5
- genesis_entry_footer genesis_entry_footer_markup_close p:15
- genesis_after_entry genesis_do_author_box_single p:8
- genesis_after_entry genesis_adjacent_entry_nav
- genesis_after_entry genesis_get_comments_template
- genesis_before_post_title genesis_do_post_format_image
- genesis_post_title genesis_do_post_title
- genesis_post_content genesis_do_post_title
- genesis_post_content genesis_do_post_image
- genesis_post_content genesis_do_post_content
- genesis_post_content genesis_do_post_permalink
- genesis_post_content genesis_do_post_content_nav
- genesis_before_post_content genesis_post_info
- genesis_after_post_content genesis_post_meta
- genesis_after_post genesis_do_author_box_single
- genesis_loop_else genesis_do_noposts
- genesis_after_endwhile genesis_posts_nav
- do action
genesis_reset_loops
- add actions
3.24.2.6. loops.php wp:genesis:loops
genesis_do_loop()genesis_standard_loop()genesis_before_while- nothing
- (no term)
- Run while loop
the_post();genesis_before_entry- nothing
genesis_markup- open
<article class="entry">, context:entry - (no term)
- do action
genesis_entry_headerbootstrap-for-genesistheme moves gensis_do_post_image fromgenesis_entry_contentto here at p:0genesis_do_post_format_image()at p:4genesis_entry_header_markup_open()at p:5
- (no term)
<header class="entry-header">genesis_attrcontext:entry-headergenesis_do_post_title()
- (no term)
- filter
genesis_post_title_text( get_the_title() ) - (no term)
- filter
genesis_link_post_title( true ) genesis_markup- open/close
<a class="entry-title-link">context:entry-title-link - (no term)
- filter
genesis_entry_title_wrap genesis_markup- open/close
$wrap<h1 class="entry-title"></h1>, content:~$title~ context:entry-title - (no term)
genesis_post_title_output( $output, $wrap, $title)genesis_post_info()at p:12- byline
- (no term)
- filter
genesis_post_info$filtered = genesis_post_info( '[post_date]'.__( 'by', 'genesis'). '...' )do_shortcode()p:20
genesis_markup- if post type supports genesis-entry-meta-before-content, open/close
<p class="entry-meta">content:genesis_strip_p_tags( $filtered )context:entry-meta-before-content- apply filter
genesis_attr_entry-meta-before-contentgenesis_attributes_entry_meta()- change class to
entry-meta
- close
</header>
- apply filter
genesis_before_entry_content- nothing
- (no term)
<div class="entry-content">genesis_markup- context:entry-content
- (no term)
- do action
genesis_entry_contentgenesis_do_post_image()at p:8 wp:genesis:structure:post.php
- (no term)
bootstrap-for-genesistheme moves this togenesis_entry_header- (no term)
- If not singular and wp:genesis:options has
content_archive_thumbnail, display the image - (no term)
$img = genesis_get_image()genesis_markup- open/close
<a class="entry-image-link"></a>- content:
wp_make_content_images_responsive( $img ) - context:entry-image-link
genesis_do_post_content()wp:genesis:structure:post.phpgenesis_do_post_content_navat p:12 wp:genesis:structure:post.phpgenesis_do_post_permalinkat p:14 wp:genesis:structure:post.php
- content:
- (no term)
- close
</div>entry-content genesis_after_entry_content- nothing
- (no term)
genesis_entry_footer<footer class="entry-footer">genesis_entry_footer_markup_openat p:5Filed under ...</footer>genesis_entry_footer_markup_closeat p:15
genesis_markup- close
</arcitle>, context:entry - (no term)
genesis_after_entrygenesis_do_author_box_singleat p:8genesis_add_id_to_global_excludeat p:9genesis_after_entry_widget_areagenesis_adjacent_entry_navgenesis_get_comments_template()
- (no term)
- wp:f:comments_template
- (no term)
- refer to wp:genesis:comments.php
- (no term)
genesis_after_endwhilegenesis_posts_nav()div.archive-paginationgenesis_numeric_posts_nav()orgenesis_prev_next_posts_nav()
genesis_loop_else- if no post
genesis_custom_loop( $args = array() )- filter
$args = genesis_custom_loop_args $wp_query = new WP_Query( $args )genesis_standard_loop()wp_reset_query()
- filter
genesis_grid_loop( $args = array() )- filter
genesis_grid_loop_args - remove actions
genesis_before_post_titlegenesis_do_post_format_imagegenesis_post_contentgenesis_do_post_imagegenesis_post_contentgenesis_do_post_contentgenesis_post_contentgenesis_do_post_cotent_navgenesis_entry_headergenesis_do_post_format_imagegenesis_entry_contentgenesis_do_post_imagegenesis_entry_contentgenesis_do_post_contentgenesis_entry_contentgenesis_do_post_content_navgenesis_entry_contentgenesis_do_post_permalink
- add filter
post_classgenesis_grid_loop_post_class() - add action
genesis_post_contentgenesis_grid_loop_content - add action
genesis_entry_contentgenesis_grid_loop_content genesis_standard_loop()genesis_reset_loops()- remove filter
post_classgenesis_grid_loop_post_class() - remove action
genesis_post_contentgenesis_grid_loop_content() - remove action
genesis_entry_contentgenesis_grid_loop_content()
- filter
3.24.2.7. comments.php
3.24.2.8. sidebar.php
3.24.2.9. archive.php
3.24.2.10. search.php
3.24.3. Life cycle
Child theme loads its functions.php before parent genesis theme's functions.php is loaded which is the parent's init.php
- Action
genesis_pre - nothing
- (no term)
- Action
genesis_init- load textdomain
genesis_i18n()- load theme support
genesis_theme_support()wp:genesis:theme support- load post type support
genesis_post_type_support()- constants
genesis_constants()- load frameworks
genesis_load_framework()do_action( 'genesis_pre_framework' );- lib/framework.php
function genesis(){}
- lib/classes/*
- lib/functions/* wp:genesis:functions
- lib/shortcodes/* wp:genesis:shortcodes
- lib/structure/*
- lib/admin/*
- lib/js/load-scripts.php
- lib/css/load-styles.php
- lib/widgets/*
- (no term)
- Action
genesis_setupwp:genesis:action:genesis_setup- may
add_image_size - wp:f:add_image_size
- create initial layout
genesis_create_initial_layouts()wp:genesis:functions:layout.php- register default widget areas
genesis_register_default_widget_areas()- 3 sidebars are registered
header-right,sidebar,sidebar-alt- (no term)
- wp:action:after_setup_theme
- register 3 default sidebars
_genesis_register_default_widget_areas_cb()
- register 3 default sidebars
- may
3.24.4. Actions and filters
https://genesistutorials.com/visual-hook-guide/
function genesis(){}
get_header()wp:genesis:header.php- open
div, context:content-sidebar-wrap - nothing
- open
<main class="content">, context:content genesis_before_loopgenesis_do_breadcrumbs()genesis_attr( 'breadcrumb' )- choose one function to display. If not found, use
genesis_breadcrumb()- filter
genesis_breadcrumb_args
- filter
genesis_do_cpt_archive_title_description()wp:genesis:do_cpt_archive_title_description- If it's archive page and post type supports archive, then display title for archive page
- If post type supports
genesis-cpt-archives-settingsfrom wp:genesis:post type supports - filter
genesis_cpt_archive_intro_text_output( $intro_text ) - do action
genesis_archive_title_descriptions( $heading, $intro_text, 'cpt-archive-description' )genesis_do_archive_headings_open()at p:5
- If $heading or $intro_text, open
<div class="cpt-archive-description">context:cpt-archive-descriptiongenesis_do_archive_headings_headline()at p:10
- ">
genesis_attr( 'archive-title' )genesis_do_archive_headings_intro_textat p:12- echo $intro_text
- (no term)
genesis_do_archive_headings_closeat p:15
- context:cpt-archive-description
genesis_do_date_archive_title()- If is date archive
is_date()then run - do action
genesis_archive_title_description( $heading, '', 'date-archive-description')
- If is date archive
genesis_do_blog_template_heading()genesis_attr( 'blog-template-description' )genesis_do_post_title()genesis_archive_title_description
genesis_do_posts_page_heading()genesis_do_taxonomy_title_description()at p:15genesis_do_author_title_description()at p:15genesis_do_author_box_archive()at p:15genesis_do_search_title()
genesis_loopgenesis_do_loop- wp:genesis:loops
genesis_404- in template 404.php
- nothing
- close
</main>context:main genesis_after_contentgenesis_get_sidebar()get_sidebar()
- close
</div>, context:content-sidebar-wrap genesis_after_content_sidebar_wrapgenesis_get_sidebar_alt()get_sidebar( 'alt' )
get_footer()wp:genesis:footer.php
3.24.4.1. header.php wp:genesis:header.php
wp:f:get_header loads the parent theme's header.php unless child theme has header.php
genesis_doctypegenesis_titlegenesis_meta</head> <body>genesis_canonical- p:5
- (no term)
genesis_load_favicon- (no term)
genesis_do_meta_pingback- (no term)
genesis_paged_rel- (no term)
genesis_meta_name- (no term)
genesis_meta_url- (no term)
genesis_header_scripts- filter
genesis_header_scripts( genesis_get_option( 'header_scripts'))
- filter
- (no term)
genesis_custom_header_style- (no term)
- remove actions
- wp_generator
- adjacent_posts_rel_link_wp_head
- wlwmanifest_link
- wp_shortlink_wp_head
- feed_links_extra
- rel_canonical
- open
<body class="body">, context:body genesis_beforegenesis_skip_links()priority 5
- open
div, context:site-container genesis_before_headergenesis_skip_links()p:5
- do action
genesis_headergenesis_header_markup_open()atp:5context:site-header<header class="site-header">genesis_structural_wrap( 'header' )context:headergenesis_attr( 'structural-wrap' )- filter
genesis_attr_structural-wrap - filter
genesis_attr_structural-wrap_output - filter
genesis_structural_wrap-header
- (no term)
genesis_do_header()atp:10genesis_markup- open, context:title-area
genesis_site_titlegenesis_seo_site_title()
genesis_site_descriptiongenesis_seo_site_description()
genesis_markup- close, context:title-area
genesis_markup~- open, context:header-widget-area
genesis_header_right- nothing
- add filter
genesis_header_menu_args - add filter
genesis_header_menu_wrap dynamic_sidebar( 'header-right' )- (no term)
- remove_filters
genesis_markup- close, context:header-widget-area
genesis_header_markup_closeatp:15- close
</header>genesis_structural_wrap( 'header', 'close' )- context:site-header
</header>
- (no term)
- genesis_site_title
- (no term)
- genesis_site_description
- (no term)
- genesis_header_right
- (no term)
- genesis_site_description
genesis_after_header- open
div, context:site-inner genesis_structural_wrap( 'site-inner' )
3.24.4.2. genesis_standard_loop(){}
- do_action( 'genesis_before_while' );
- do_action( 'genesis_before_entry' );
- do_action( 'genesis_entry_header' );
- do_action( 'genesis_before_entry_content' );
- do_action( 'genesis_entry_content' );
- do_action( 'genesis_after_entry_content' );
- do_action( 'genesis_entry_footer' );
- do_action( 'genesis_after_entry' );
- do_action( 'genesis_after_endwhile' );
3.24.4.3. footer.php wp:genesis:footer.php
wp:f:get_footer loads parent theme's footer.php unless child theme has footer.php
genesis_structural_wrap( 'site-inner', 'close')- close
</div>, context:site-inner genesis_before_footergenesis_footer_widget_areas()dynamic_sidebar( 'footer-'. $counter )- open/close
<div class="footer-widget-area">, context:footer-widget-area, content:$widgets genesis_structural_wrap( 'footer-widgets', 'open', 0)genesis_structural_wrap( 'footer-widgets', 'close', 0)- open/close
<div class="footer-widgets">, context:footer-widgets
genesis_footergenesis_footer_markup_openp:5genesis_markup- open
<footer class="site-footer">, context:site-footer - (no term)
genesis_structurual_wrap( 'footer', 'open' )
genesis_do_footer- filter
genesis_footer_backtotop_text - nothing
- filter
genesis_footer_creds_text $creds_textnothing- (no term)
- filter
genesis_footer_output- ::
do_shortcode()p:20
- ::
- filter
genesis_footer_markup_closep:15genesis_structural_wrap( 'footer', 'close' )- close
</footer>, context:site-footer
- close
</div>, context:site-container - nothing
wp_footer()- close
</body>, context:body
3.24.4.4. comments.php wp:genesis:comments.php
genesis_before_comments- nothing
- (no term)
genesis_commentsgenesis_do_comments()genesis_markup- open
<div class="entry-comments">, context:entry-comments - (no term)
- filter
genesis_title_comments - (no term)
- actiion
genesis_list_commentsgenesis_default_list_comments()
- (no term)
- filter
genesis_comment_list_args - (no term)
wp_list_comments( $args )- (no term)
- prev link
genesis_prev_comments_link_text - (no term)
- next link
genesis_next_comments_link_text genesis_markup- open/close
div, content:$pagination, context:comments-pagination genesis_markup- close
div, context:entry-comments
genesis_after_comments- none
genesis_before_pings- none
- (no term)
genesis_pingsgenesis_do_pings()- filter
genesis_attr_entry-pings - open
div, context:entry-pings genesis_list_pings- close
div, context:entry-pings
- filter
genesis_after_pings- none
genesis_before_comment_form- none
- (no term)
genesis_comment_formgenesis_do_comment_form()
genesis_after_comment_form- none
3.24.5. Theme support wp:genesis:theme support
- https://sridharkatakam.com/comprehensive-guide-genesis-theme-supports/
- Refer to wp:f:add_theme_support
- bool wp:genesis:theme support:genesis-after-entry-widget-area
- array wp:genesis:theme support:genesis-structural-wraps
Specify contexts and add an HTML wrapper as a child element under each context element
// to remove a named HTML element add_theme_support( 'genesis-structural-wraps', array( 'header', 'menu-primary', 'menu-secondary', // 'site-inner', 'footer-widgets', 'footer' ) );
Filter contexts using
genesis_theme_support_structural_wraps- Wrapper is inserted by wp:genesis:functions:genesis_structural_wrap
- array wp:genesis:theme support:genesis-accessibility
- headings
- bool whether to wrap content with
<hn></hn> - (no term)
- skip-links
- drop-down-menu
- genesis_superfish_enabled to determine whether to use superfish
- screen-reader-text
3.24.6. Post Type Supports wp:genesis:post type supports
Refer to wp:f:add_post_type_support
- genesis-seo
- enable seo fields on wp-admin
- genesis-scripts
- add a field for per-post script on wp-admin
- genesis-layouts
- default layout for a CPT
genesis-cpt-archives-settings- enable archive settings
- After enabled,
Archive Settingsappears on wp-admin for that post type - if empty, return post type's label name
- Refer to wp:genesis:do_cpt_archive_title_description
- After enabled,
- genesis-entry-meta-before-content
- disable
genesis_post_info(byline) - genesis-entry-meta-after-content
- disable
<footer></footer>andgenesis_post_meta() - genesis-after-entry-widget-area
- disable
<div class="after-entry widget-area"></div> - (no term)
- genesis-adjacent-entry-nav
3.24.7. Shortcodes wp:genesis:shortcodes
All shortcodes have filters
[footer_home_link after=""][footer_loginout after="" before="" redirect=""][footer_backtotop before="" after="" href="#wrap" nofollow="true" text="return to top of page"]- If
genesis_html5(), then empty
- If
[footer_copyright before="" after="" copyright="©"]sprintf( '[footer_copyright before="%s "] · [footer_loginout]', __( 'Copyright', 'genesis' ) );
[footer_home_link before="" after="" text="get_bloginfo( 'name' )"]
3.24.8. Child theme
functions.php
add_action( 'genesis_setup', 'bfg_childtheme_setup', 15 ); function bfg_childtheme_setup() { // Load parent Genesis theme init.php include_once( get_template_directory() . '/lib/init.php' ); // ... }
3.24.9. Genesis Visual Hook Guide
3.25. Customize API wp:api:customize
3.25.1. WP_Customize_Manager wp:WP_Customize_Manager
- Panels > Sections > Controls > Settings
- wp:action:customize_register">
do_action( 'customize_register', $wp_customize );add_setting, add_control, add_section, add_panel Refer to wp:f:get_theme_mod after add_setting
add_action( 'customize_register', function ( $wp_customize ) { $wp_customize->add_setting( 'my_setting', array( 'default' => '', // Still need to save/Publish value on Appearance > My Section > My Control > My Setting // 'type' => 'theme_mod', // default. Use get_theme_mod later ) ); $wp_customize->add_section( 'my_section', array( 'title' => __( 'My Section', 'mytheme' ), 'priority' => 30, // d:160 // 'panel' => '', // default ) ); $wp_customize->add_control( new WP_Customize_Control( $wp_customize, 'my_control_for_my_setting', // or simply use my_setting array( 'label' => __( 'My Control Title', 'theme_name' ), 'section' => 'my_section', 'settings' => 'my_setting', 'type' => 'text', ) ) ); } );
3.26. Admin
3.26.1. Toolbar - Admin bar wp:admin:toolbar
- Admin bar is replaced with Toolbar since WP 3.3
wp_footer();is called.<body>will have classadmin-barand/wp-includes/css/admin-bar.min.cssis addedshow_admin_bar( false );is not run in functions.php- In newer version of wp, wp:filter:show_admin_bar can be used to turn off the admin bar
If you have sticky or position:fixed element .sticky-header
.sticky-header { position: fixed; top: 0; } .admin-bar .sticky-header { top: 46px; /* 0 + 46 */ } @media screen and (min-width: 783px) { .admin-bar .sticky-header { top: 32px; /* 0 + 32 */ } }
3.26.2. List Table API WP_Posts_Lists_Table wp:api:list table
wp-admin/class-wp-posts-list-table.php
3.26.2.1. Action manage_${post_type}_posts_custom_column wp:api:list table:action:manage_postspost_type_custom_column
do_action( "manage_{$post->post_type}_posts_custom_column", $column_name, $post->ID );- echo column values
- For native post types
- manage_posts_custom_column
- manage_pages_custom_column
- No!!! manage_users_custom_column is a filter not an action!
- manage_comments_custom_column
- manage_media_custom_column
- manage_plugins_custom_column
- manage_themes_custom_column
- manage_link_custom_column
- manage_sites_custom_column
- Populate or modify column values
add_action( 'manage_book_posts_custom_column', function ( $column, $post_id ) { switch ( $column ) { case 'book_author' : $terms = get_the_term_list( $post_id, 'book_author', '', ',', '' ); if ( is_string( $terms ) ) { echo $terms; } else { _e( 'Unable to get author(s)', 'your_text_domain' ); } break; case 'publisher' : echo get_post_meta( $post_id, 'publisher', true ); // custom post meta // echo get_post_meta( $post_id, '_custom_post_order', true) // ACF plugin // echo get_field('custom_order', $post_id); break; } }, 10, 2 );
3.26.2.2. Filter manage_${taxonomy}_custom_column and manage_users_custom_column wp:api:list table:filter:managetaxonomy_custom_column
- Filter column values
- Refer to wp:api:taxonomy
3.26.2.3. Filter manage_${post_type}_posts_columns wp:api:list table:filter:managepost_type_posts_columns
- Add/remove columns
- Almost the same as wp:api:plugin:filter:managescreen_id_columns, but this one runs after
add_filter( 'manage_book_posts_columns', function ( $columns ) { // var_dump('CPT book manage columns filter'); var_dump($columns); // unset( $columns['author'] ); // don't show a column $columns['book_author'] = __( 'Author', 'your_text_domain' ); // add a column $columns['publisher'] = __( 'Publisher', 'your_text_domain' ); // add another column return $columns; } );
3.26.2.4. Filter manage_{screen_id}_columns wp:api:list table:filter:managescreen_id_columns
- e.g.
apply_filters( 'manage_posts_columns', $posts_columns, $post_type ); - Almost the same as wp:api:plugin:filter:managepost_type_posts_columns
- screen_id
edit-my_tax_name- refer to wp:api:plugin:action:manage_postspost_type_custom_column
3.26.2.5. Filter manage_{screen_id}_sortable_columns wp:api:list table:filter:managescreen_id_sortable_columns
apply_filters( "manage_{$this->screen->id}_sortable_columns", $sortable_columns );- Refer to wp:api:taxonomy
- screen_id
edit-my_tax_name- refer to wp:api:plugin:action:manage_postspost_type_custom_column
3.27. Custom Cache
Same page request custom cache, Single request cache
function lili_request_cache($field, $set = null) { static $custom_cache = []; if ($set !== null) { $custom_cache[$field] = $set; } elseif ( ! isset($custom_cache[$field]) || $custom_cache[$field] == null) { // Set default by field switch ($field) { case 'c': $custom_cache[$field] = 'default value for c'; break; default: $custom_cache[$field] = null; break; } } return $custom_cache[$field]; } lili_request_cache('a', 1); var_dump('a: ', lili_request_cache('a')); // 1 var_dump('b: ', lili_request_cache('b')); // null var_dump('a: ', lili_request_cache('a')); // 1 lili_request_cache('a', 2); var_dump('a: ', lili_request_cache('a')); // 2 var_dump('c: ', lili_request_cache('c')); // default value for c
3.28. Custom page
3.29. Database wp:db
3.29.1. wp_posts wp:db:wp_posts
- ID
- bigint(20)
- post_author
- bigint(20)
- post_date
- datetime
- post_date_gmt
- datetime. Refer to wp:post:post_date_gmt
- post_content
- longtext
- post_title
- text
- post_excerpt
- text
- post_status
- varchar(20) d:'publish' Refer to wp:f:register_post_status
- post_name
- varchar(200) d:'' name in slug
- to_ping
- text
- ping_status
- varchar(20) d:'open'
- pinged
- text
- post_modified
- datetime
- post_modified_gmt
- datetime
- post_content_filtered
- longtext
- post_parent
- bigint(20)
- guid
- varchar(255)
- menu_order
- int(11)
- post_type
- varchar(20) d:'post'
- Core
- post, page, attachment, nav_menu_item, revision, oembed_cache wp:api:oembed
- (no term)
- Custom post type slug
- post_mime_type
- vchar 100 blank for non media posts. e.g. image/jpeg
- comment_status
- varchar(20) d:'open'
- comment_count
- bigint(20)
- post_password
- varchar(20) d:''
#+NAME Number of posts per Post Type, Year, Month
SELECT -- YEAR(p.post_date) AS year, -- MONTH(p.post_date) AS month, post_type, count(*), -- ,p.ID, p.post_title, p.post_type, p.post_status 1 FROM wp_posts p WHERE 0 = 0 AND p.post_type NOT IN ('attachment', 'revision', 'nav_menu_item', 'oembed_cache') -- AND p.post_status = 'publish' GROUP BY -- YEAR(p.post_date), -- MONTH(p.post_date), post_type ORDER BY year DESC, month DESC, post_date DESC -- LIMIT 100
Get terms for posts
SELECT t.term_id, t.name as term_name, t.slug as term_slug, t.term_group, tr.term_order, tt.term_taxonomy_id, tt.taxonomy as taxonomy, tt.parent, p.ID as post_id, p.post_title, p.post_type, p.post_status, -- count(*), 1 FROM wp_posts p INNER JOIN wp_term_relationships tr ON tr.object_id = p.ID INNER JOIN wp_terms t ON t.term_id = tr.term_taxonomy_id INNER JOIN wp_term_taxonomy tt ON tt.term_id = t.term_id WHERE 1 = 1 AND p.post_type NOT IN ('attachment', 'revision', 'nav_menu_item', 'oembed_cache', 'wp-types-group') AND p.post_status = 'publish' -- AND tt.taxonomy = 'post_tag' -- AND ( t.slug LIKE 'TTNA' OR t.name LIKE 'TTNA' ) -- AND t.slug LIKE 'GCKC' -- AND p.ID = '144571' -- GROUP BY t.term_id, -- tt.taxonomy, -- 1 ORDER BY p.post_date DESC
3.29.2. wp_terms
- term_id
- int, auto_increment
- name
- varchar(200)
- slug
- varchar(200)
- term_group
- int
3.29.3. wp_term_relationships wp:db:wp_term_relationships
- object_id
- int, (e.g. ID in wp_posts), 0
- term_taxonomy_id
- int, 0
- term_order
- int, 0
#+NAME Return posts that don't have certain terms
SELECT p.ID as post_id, p.post_title, p.post_type, p.post_status, CONCAT('https://www.a.ca/wp-admin/post.php?action=edit&post=' , p.ID) FROM wp_posts p WHERE 1 = 1 AND p.post_type IN ('vendors') -- AND p.post_status IN ('publish', 'draft') AND NOT EXISTS( SELECT * FROM wp_term_relationships tr INNER JOIN wp_terms t ON t.term_id = tr.term_taxonomy_id INNER JOIN wp_term_taxonomy tt ON tt.term_id = t.term_id WHERE tr.object_id = p.ID AND tt.taxonomy = 'vendor_city' AND t.slug IN ('slug1', 'slug2') ) ORDER BY p.post_date DESC
3.29.4. wp_term_taxonomy wp:db:wp_term_taxonomy
- term_taxonomy_id
- bigint(20)
- term_id
- bigint(20)
- taxonomy
- varchar(20) category, post_tag, link_category, nav_menu, ngg_tag
- description
- longtext
- parent
- bigint(20)
- count
- bigint(20) wp:db:wp_term_taxonomy:count
3.29.5. wp_termmeta
- meta_id
- bigint(20) unsigned, PK, start from 1
- term_id
- bigint(20) unsigned, IND, default 0
- meta_key
- varchar(255) e.g.
wpcf-publication_date - meta_value
- longtext
3.29.6. wp_postmeta wp:db:wp_postmeta
- meta_id PK
- bigint 20
- post_id
- bigint 20
- meta_key
- vchar(255) if custom field has prefix
_, It means the field is private and does not display on Post Edit page on wp-admin- wp:plugin:acf key
custom_orderwith value from wp-admin and_custom_orderwith valuefield_ALPHANUMERICRANDOMwhich refers tablewp_postswithpost_name = 'field_ALPHANUMERICRANDOM' wp:custom page template e.g.
page-news.phpdefaultSELECT pm.meta_value, COUNT(*) -- ,p.id, p.post_title, p.post_type, p.post_parent, p.post_date, p.guid, p.post_name FROM wp_posts p INNER JOIN wp_postmeta pm ON pm.post_id = p.id WHERE 1=1 AND p.post_status = 'publish' AND pm.meta_key = '_wp_page_template' -- AND p.ID = '126126' -- AND pm.meta_value = 'page-no-title.php' GROUP BY pm.meta_value ORDER BY p.post_date DESC
unix_timestamp:useride.g.1566567414:38useride.g.38
- wp:plugin:acf key
- meta_value
- longtext string or php serialized string
#+NAME Get all meta keys and values for a post
SELECT p.post_type, p.ID, pm.meta_key, pm.meta_value, pm.meta_id FROM wp_posts p INNER JOIN wp_postmeta pm ON pm.post_id = p.id WHERE 1=1 AND p.ID = '123'
#+NAME Get all posts with a meta key
SELECT p.post_type, p.ID, pm.meta_key, pm.meta_value, pm.meta_id -- ,CAST(pm.meta_value AS Date) FROM wp_posts p INNER JOIN wp_postmeta pm ON pm.post_id = p.id WHERE 1=1 -- AND p.post_type = 'agenda' AND p.post_status = 'publish' AND pm.meta_key LIKE '%agenda_date%' ORDER BY p.post_date DESC
3.29.6.1. meta_key Media files
- _wp_attached_file
2009/11/a.jpg2014/02/a.pdfcombine with wp:db:wp_options:upload_path wp:db:wp_postmeta:_wp_attached_file- _wp_attachment_image_alt
- string
- _thumbnail_id
- int, this is the featured image post id (123) of parent post (122)
- Featured image post 123 with post_parent = the real parent post (122)
- Featured image post 123 also has these meta keys and values described here
- Featured image post 123 has post_type as attachment
- (no term)
_wp_attachement_backup_sizes
Array ( [full-orig] => Array ( [width] => 1024 [height] => 768 [file] => Tree.jpg ) [full-1317235120383] => Array ( [width] => 316 [height] => 237 [file] => Tree-e1317235110739.jpg ) )_wp_attachment_metadataserialized string. Only images have non-empty array. It contains all wp:f:add_image_size for a featured image
Array ( [width] => 100 [height] => 100 [hwstring_small] => height='96' width='96' [file] => 2009/11/competitiveedge_091109_tease.jpg [sizes] => Array ( [thumbnail] => Array ( [file] => competitiveedge_091109_tease-100x60.jpg [width] => 100 [height] => 60 ) ) [image_meta] => Array ( [aperture] => 0 [credit] => [camera] => [caption] => [created_timestamp] => 0 [copyright] => [focal_length] => 0 [iso] => 0 [shutter_speed] => 0 [title] => ) )
3.29.7. wp_options wp:db:wp_options
- Refer to wp:options
- bigint(20) unsigned
- varchar(191) d:''
- upload_path
wp-content/uploadswp:db:wp_options:upload_path- rewrite_rules
- wp:db:wp_options:rewrite_rules
- active_plugins
- array of plugins' php files loaded in order wp:db:wp_options:active_plugins
- siteurl
https://www.a.cawp:db:wp_options:siteurl
- longtext
- varchar(20) d:'yes'
3.30. Plugins
3.30.1. Custom Plugin
- https://codex.wordpress.org/Plugin_Resources
- https://codex.wordpress.org/Writing_a_Plugin
- https://developer.wordpress.org/plugins/the-basics/#getting-started
- https://github.com/DevinVinson/WordPress-Plugin-Boilerplate
- https://github.com/levonlee/gridli-vue
- Use hyphen for plugin name and folder
wp_content/plugins/li-custom/li-custom.php - readme.txt
- Use wp:action:activated_plugin to change plugin loading order
3.30.2. GitHub Updater wp:plugin:github-updater
- Update and install GitHub, Bitbucket, GitLab and Gitea hosted plugins and themes
- https://github.com/afragen/github-updater
- Installation
- just download the zip
3.30.3. Classic Editor wp:plugin:classic-editor
3.30.5. Translation
3.30.5.1. WPML wp:plugin:sitepress-multilingual-cms wp:plugin:wpml
- Before there's lifetime license but now only yearly
- 3 levels of license: Multilingual Blog, CMS and Agency. Agency can have unlimited website registration which provides plugin update notification
- Working with themes and ohter plugins
- Change language flag Languages > Site Languages > Edit languages
- pluginfolder/res/flags
- size 18x12
- Settings > Taxonomy Translation
- Translatable - only show translated items (when you edit a single post)
- Taxonomy translation
- copy to all languages
- copy default language setting to other languages
- (no term)
- Refresh permalinks after
- Translate Widgets
Requires String Translation addon. Change domain to Widgets.
- May also change widget variable
- Go to the bottom of String Translation and click Translate texts in admin screens
- https://wpml.org/documentation/getting-started-guide/translating-widgets/
- May also change widget variable
- Template
- Get languages
- https://wpml.org/wpml-hook/wpml_active_languages/
- call after wp:action:wp
$languages = apply_filters( 'wpml_active_languages', NULL, 'orderby=id&order=desc' );
- Homepage URL
<a href="<?php echo apply_filters( 'wpml_home_url', get_option( 'home' ) ); ?>">Home</a>
- Manual update plugin
- Deactivate it and other addons
- Copy folder
- Activate
- shortcodes
- wpml-string
Requires WPML String Translation
[wpml-string context="my-domain" name="my-name"]My string[/wpml-string]To translate this string, go to the WPML -> String Translation page and use the following info (replacing the names with your own):- Domain: my-domain
- Name: my-name
- String: My string
Pay attention to the page when the string is registered. If the string is first registered in French, then you should
Change the language of selected stringsto French.
- wpml-string
3.30.5.2. polylang wp:plugin:polylang
https://polylang.wordpress.com/documentation/
Manage multilingual posts in one post per language (e.g. WPML - paid, xili-language, Polylang, Bogo or Sublanguage). Translations are then linked together, indicating that one page is the translation of another.
There are other types of multilingual plugins:
- Store all languages alternatives for each post in the same post (e.g. qTranslate-X, WPGlobus).
- Manage translations on the generated page instead of using a post context (e.g. Transposh and Global Translator).
- Plugins like Multisite Language Switcher, Multilingual Press, and Zanto, link together separate WordPress network (multisite) installations for each language by pinging back and forth.
Add languages including English (the first language) Click on a post/page, click on + for another language and then create the translation page. You can select another existing page and add that page to the current page's translation. Posts in different languages will be created with different URLs. Go to edit one post and you can go to different translated pages.
URLs
/sample-page /zh/chinese-sample-page
You can create a different menu for each language
Determine the locale of current page
//get_locale() :: 'en_US' //get_bloginfo('language') :: 'en-US' // in template <div class="nav-previous alignleft"> <?php $currentlang = get_bloginfo('language'); if($currentlang=="en-US"): ?> This is English <?php else: ?> This is Spanish <?php endif; ?> </div>
Display links to posts translations within the loop
<?php while ( have_posts() ) : the_post(); ?> <ul class='translations'><?php pll_the_languages( array( 'post_id' => $post->ID ) ); ?></ul> <?php the_content(); ?> <?php endwhile; ?>
Other functions https://polylang.wordpress.com/documentation/documentation-for-developers/functions-reference/
<?php // outputs a list of languages names ?> <ul><?php pll_the_languages(); ?></ul> <?php // outputs a flags list (without languages names) ?> <ul><?php pll_the_languages(array('show_flags'=>1,'show_names'=>0)); ?></ul> <?php // outputs a dropdown list of languages names ?> <?php pll_the_languages(array('dropdown'=>1)); ?>
- String translation is possible
There're 3 ways to register a string to translate
- pll_register_string($name, $string, $group, $multiline);
- $name => (required) name provided for sorting convenience (ex: ‘myplugin’)
- $string => (required) the string to translate
- $group => (optional) the group in which the string is registered, default is ‘polylang’
- $multiline => (optional) if set to true, the translation text field will be multiline, default is false
function register_strings() { pll_register_string('mytheme', 'some_string'); } add_action('init','register_strings'); // in template to echo pll_e('some_string'); // to return pll__('some_string'); // return in another language that might not be the current language pll_translate_string($string, $lang);
- WPML API
for example icl_register_string($context, $name, $string);
- wpml-config.xml
Some plugin or theme might include this WPML plugin xml in its root directory. https://polylang.pro/doc/the-wpml-config-xml-file/
<wpml-config> <custom-fields> <custom-field action="copy">quantity</custom-field> <custom-field action="translate">custom-title</custom-field> </custom-fields> <custom-types> <custom-type translate="1">book</custom-type> <custom-type translate="1">DVD</custom-type> </custom-types> <taxonomies> <taxonomy translate="1">genre</taxonomy> </taxonomies> <admin-texts> <key name="my_plugins_options"> <key name="option_name_1" /> <key name="option_name_2" /> <key name="options_group_1"> <key name="sub_option_name_11" /> <key name="sub_option_name_12" /> </key> </key> <key name="simple_string_option" /> </admin-texts> </wpml-config>
custom-fields: the custom field name needs to be provided and also the action
- ignore
- no action
- translate
- value is copied from the source but may be modified
- copy
- value is copied from the source and synchronized across translations
custom-types: the custom post types that Polylang should manage. Once translate attribute is set, it cannot be changed in Polylang settings. 1 is to manage, 0 is not to manage.
taxonomies: the custom tax that Polylang should manage.
admin-texts: when get_option is called in themes or plugins, Polylang can filter these calls and provide translation to the values of these options.
- Add a key name. A key can have children for serialization.
- pll_register_string($name, $string, $group, $multiline);
- Sitemap and Permalink
- After adding new language, the sitemap will be empty including the default language
- wp:plugin:all-in-one-seo-pack has to have Polylang set to "The language is set from different domains"
- originaldomain.com/page1
- fr.originaldomain.com/page1-french-name
- Even the domain is different, the permalinks of the 2 pages in 2 different languages have to be different.
- originaldomain.com/sitemap.xml
- fr.originaldomain.com/sitemap.xml
3.30.6. Image
3.30.6.1. Resize image on the fly fly-dynamic-image-resizer
$image = fly_get_attachment_image_src( get_post_thumbnail_id(), array(160, 9999), false // true to crop to exact dimension ); $image :: array( 'src', 'width', 'height' ); fly_add_image_size('lili-w600', 600, 9999, true); $image = fly_get_attachment_image($post->id, 'lili-w600'); // HTML
3.30.6.2. Imsanity - Resize Image wp:plugin:imsanity
- I think gif will not be included which is good
- Resize upload images
- If
Convert PNG to JPGis No, then PNG file will not be resized automatically - Batch processing existing images
- Some image files may fail. Don't select them
- May use global variable to change the default 250 images returned for batch processing
3.30.6.3. Crop-Thumbnails wp:plugin:crop-thumbnails
- Resize featured image, Crop featured image, crop thumbnail
3.30.6.4. multiple-featured-images wp:plugin:multiple-featured-images
add_filter( 'kdmfi_featured_images', function( $featured_images ) { $args_1 = array( 'id' => 'featured-image-2', 'desc' => 'Your description here.', 'label_name' => 'Featured Image 2', 'label_set' => 'Set featured image 2', 'label_remove' => 'Remove featured image 2', 'label_use' => 'Set featured image 2', 'post_type' => array( 'page' ), ); $args_2 = array( 'id' => 'featured-image-3', 'desc' => 'Your description here.', 'label_name' => 'Featured Image 3', 'label_set' => 'Set featured image 3', 'label_remove' => 'Remove featured image 3', 'label_use' => 'Set featured image 3', 'post_type' => array( 'page', 'post' ), ); $featured_images[] = $args_1; $featured_images[] = $args_2; return $featured_images; });
3.30.7. Image, Media in cloud
3.30.7.1. S3-Uploads wp:plugin:S3-Uploads
- https://github.com/humanmade/S3-Uploads
composer require humanmade/s3-uploadsor download and extractmanual-install.zipwp-config.phpdefine( 'S3_UPLOADS_BUCKET', 'media.a.com' ); define( 'S3_UPLOADS_KEY', 'xxx' ); define( 'S3_UPLOADS_SECRET', 'yyy' ); define( 'S3_UPLOADS_REGION', 'us-east-1' ); define( 'S3_UPLOADS_BUCKET_URL', 'https://media.a.com' ); // without trailing slash. If not defined, file urls in db will start with [bucketname].s3.amazonaws.com/uploads/[file path] define( 'S3_UPLOADS_AUTOENABLE', false ); // True to redirect all existing media files in db to `s3://`. Before doing that, upload all existing files to S3 first.
wp plugin activate S3-Uploadswp s3-uploads verifywp s3-uploads create-iam-user --admin-key=<key> --admin-secret=<secret>- If you want to create IAM user yourself, use this policy
wp s3-uploads generate-iam-policy
wp s3-uploads ls [<path>]wp s3-uploads upload-directory wp-content/uploads/ uploads- May use
--syncand--dry-run
- May use
wp s3-uploads cp ./text.txt s3://mybucket/test.txt- use aws:cloudfront to avoid setting the following
- Expire in 30 days
define( 'S3_UPLOADS_HTTP_CACHE_CONTROL', 30 * 24 * 60 * 60 );- On top, set
Expiresheader e.g. to set an asset to not expire define( 'S3_UPLOADS_HTTP_EXPIRES', gmdate( 'D, d M Y H:i:s', time() + (10 * 365 * 24 * 60 * 60) ) .' GMT' );e.g. expire in 10 years
define( 'S3_UPLOADS_DISABLE_REPLACE_UPLOAD_URL', true );define('S3_UPLOADS_OBJECT_ACL', 'private');- Different endpoints e.g. Digital Ocean Spaces
- Local dev
- Disable the plugin or
- mock S3 with a local stream wrapper and actually store the uploads in
wp-content/uploads/s3/
3.30.7.2. WP Offload Media (formerly WP Offload S3) wp:plugin:amazon-s3-and-cloudfront
- Free version
- Cloud storage providers supported: S3, DigitalOcean Spaces, Google Cloud Storage
- Only newly uploaded files will be copied to the cloud
- Pro version: https://deliciousbrains.com/wp-offload-media/
- Plans
- Bronze: 2k offloaded media items across unlimited sites
- Silver: 6k
- Gold: 20k, multisite, all integrations with addons $199/yr
- Platinum: 40k, and Gold
- Adamantium: 100k, and Platinum
Offloaded media item is 1 item in Media Library with all resized images
- If exceeded, these will be disabled
- Offload of existing media library tool
- Download to server tool
- Remove from server tool
- On demand bucket actions for attachments (Copy to Bucket or Remove from Server in Media page)
- WooCommerce integration for new product files
- New files set to private for WooCommerce
- New files set to private for wp:plugin:easy-digital-downloads
- treat the Plan difference as the initial upload size of WP website
- Upload existing Media Library
- Assets Pull Addon: serve CSS, JS, images, fonts from CloudFront. Available with Gold license
- wp:plugin:acf, wp:plugin:woocommerce, wp:plugin:wpml, wp:plugin:easy-digital-downloads, wp:plugin:metaslider
- Plans
3.30.8. User, Roles, Capability
3.30.8.1. User Role Editor wp:plugin:user-role-editor
- Add or remove capapbilities to a role
- Add or remove capabilities to a user
3.30.8.2. Capability Manager Enhanced wp:plugin:capability-manager-enhanced
- Create Roles and assign capabilities
3.30.9. Fields
3.30.9.1. ACF wp:plugin:advanced-custom-fields wp:plugin:acf
- Add a field group and then add custom fields under the group.
- Use
Locationto define rules to assign the field group to a custom post type - Hide the default post fields (Content editor, comments, categories, etc.)
- Make custom fields required
- wp:The_Loop, to display a custom field">
the_field('field_name') get_field('field_name');
- Image field
- Post object field
echo '<pre>'; print_r( get_field('my_post_objects') ); echo '</pre>'; // single post object $post_object = get_field('my_post_object'); global $post; if( $post_object ): // override $post $post = $post_object; setup_postdata( $post ); ?> <div> <h3><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h3> <span>Post Object Custom Field: <?php the_field('field_name'); ?></span> </div> <?php wp_reset_postdata(); // IMPORTANT - reset the $post object so the rest of the page works correctly ?> <?php endif; ?> // multiple post objects $post_objects = get_field('my_post_objects'); if( $post_objects ): ?> <ul> <?php foreach( $post_objects as $post_object): ?> <li> <a href="<?php echo get_permalink($post_object->ID); ?>"><?php echo get_the_title($post_object->ID); ?></a> <span>Post Object Custom Field: <?php the_field('field_name', $post_object->ID); ?></span> </li> <?php endforeach; ?> </ul> <?php endif;
- Repeater field
<?php if ( have_rows( 'repeater_field_name' ) ): ?> <ul class="slides"> <?php while ( have_rows( 'repeater_field_name' ) ): the_row(); // vars $image = get_sub_field( 'image' ); $content = get_sub_field( 'content' ); $link = get_sub_field( 'link' ); $post_obj = get_sub_field( 'my_post_object' ); ?> <li class="slide"> <?php if ( $link ): ?> <a href="<?php echo $link; ?>"> <?php endif; ?> <img src="<?php echo $image['url']; ?>" alt="<?php echo $image['alt'] ?>"/> <?php if ( $link ): ?> </a> <?php endif; ?> <?php echo $content; ?> <?php echo get_the_title($post_obj->ID); ?> </li> <?php endwhile; ?> </ul> <?php endif; ?>
Get first row or random row
$rows = get_field('repeater_field_name' ); // get all the rows $first_row = $rows[0]; // first row $rand_row = $rows[ array_rand( $rows ) ]; // get a random row $rand_row_image = $rand_row['sub_field_name' ]; // get the sub field value // Note // $first_row_image = 123 (image ID) $image = wp_get_attachment_image_src( $rand_row_image, 'full' ); // url = $image[0]; // width = $image[1]; // height = $image[2]; ?> <img src="<?php echo $image[0]; ?>" />
- Relationship field
<?php // speaker post type has a relationship field speaker_agenda to relate to post type agenda // the following example's main query is single-speaker $posts = get_field( 'relationship_field_name' ); // returns an array of WP_Post obj if ( $posts ): ?> <ul> <?php // global $post; // Required if foreach uses $post as item foreach ( $posts as $p ): // variable must NOT be called $post (IMPORTANT) ?> <li> <a href="<?php echo get_permalink( $p->ID ); ?>"><?php echo get_the_title( $p->ID ); ?></a> <span>Custom field from $post: <?php the_field( 'author', $p->ID ); ?></span> </li> <?php endforeach; ?> </ul> <?php endif; // return an array of post ids $session_ids = get_field( 'speaker_agenda', false, false ); // first false is to use the global $post, 2nd false is not to format the value as a post obj // further sort $query = new WP_Query( array( 'post_type' => 'agenda', 'posts_per_page' => - 1, 'post__in' => $session_ids, 'post_status' => 'publish', 'orderby' => 'post__in', ) ); // reverse query. The following's main query is single agenda $speakers = get_posts( array( 'post_type' => 'speaker', 'meta_query' => array( array( 'key' => 'relationship_field_name', // name of custom relationship field 'value' => '"' . get_the_ID() . '"', // matches exactly "123", not just 123. This prevents a match for "1234" 'compare' => 'LIKE' ) ) ) ); if ( $speakers ): ?> <ul> <?php foreach ( $speakers as $speaker ): ?> <?php $photo = get_field( 'photo', $speaker->ID ); ?> <li> <a href="<?php echo get_permalink( $speaker->ID ); ?>"> <img src="<?php echo $photo['url']; ?>" alt="<?php echo $photo['alt']; ?>" width="30"/> <?php echo get_the_title( $speaker->ID ); ?> </a> </li> <?php endforeach; ?> </ul> <?php endif; ?>
- Query
if ( in_array( $query->get( 'post_type' ), array( 'event' ) ) ) { $query->set( 'orderby', 'meta_value' ); $query->set( 'meta_key', 'event_start' ); $query->set( 'order', 'ASC' ); $query->set( 'posts_per_page', '-1' ); $now = strtotime( '12:00:00' ); $meta_query = array( 'relation' => 'AND', array( 'key' => 'event_end', 'value' => date( 'Ymd'), // value in db is '20190131' for Jan 31, 2019. So the format should Ymd //'value' => date( 'Ymd', strtotime( '-12 day', $now ) ), 'compare' => '>=', ) ); $query->set( 'meta_query', $meta_query ); }
- Add a field group
3.30.9.2. Insert Headers and Footers
3.30.10. Post, Post Type
3.30.10.1. Types - Toolset Types wp:plugin:types
- WP Types is different! Use types_render_field
- Database
- Post types
- wp-types-group
- custom field group
- wp-types-term-group
- custom taxonomy group
- wp-types-user-group
- custom user field group
- Post types
- underscores can be
-based on how field is named in the pluginSame name in
meta_keyinwp_postmeta- if field is set to not-show on wp-admin,
_is appended - Real value is stored as
_wpcf-fieldname_in_toolset_ui - A multi-value field also has a field
_wpcf_fieldname_in_toolset_ui-sort-order
- The value of field
wpcf_fieldname_in_toolset_uiis''if the field has no value
- if field is set to not-show on wp-admin,
- Refer to WP Types Custom Field Checkbox and WP Meta Query
$newsletters = types_render_field('newsletters', array('output'=>'raw')); // the real field name is actually wpcf-newsletters // For single line (text) field, use 'output'=>'raw' // For WYSIWYG field, use 'output'=>'html'. 'raw' might be plain text // For Image field $_image = types_render_field('field-image', array('output'=>'raw')); // Output original full image url // If 'output' => 'raw', 'size' and 'resize' will be ignored // use 'url' => 'true' to output the url of the resized image. $args = array('output'=>'normal', 'url'=>'true', 'width'=> 300, 'resize'=>'proportional'); // Output image url of a fixed width and dynamic height $args = array('output'=>'normal', 'url'=>'true', 'size'=>'medium', 'resize'=>'proportional' ); // Output image url of a max width and height // if 'size' is set, 'width' and 'height' will be ignored /* * 'size' => 'custom_image_size' | 'full' | 'large' | 'medium' | 'thumbnail' * Set 'size' and also 'resize' => 'crop' | 'proportional' | 'stretch' | 'pad' * If resize is necessary, set 'output' => 'normal' * 'padding_color' => 'transparent' | hex when resize=>pad */ // For Image field.
3.30.10.2. Post Type Switcher wp:plugin:post-type-switcher
- Switch a post to another post type
3.30.10.3. CPT UI wp:plugin:custom-post-type-ui
Refer to wp:add cpt
- Remember to refresh permalinks
Premium Custom Post Type UI Extended can show posts of any CPT inside a post using shortcode
- Extra CPT
supportfor Genesis and other plugins - https://docs.pluginize.com/article/28-third-party-support-upon-registration
3.30.10.4. Post Expirator wp:plugin:post-expirator
- Expire posts. Change post status to one of the following:
- draft
- trash
- private
- Stick or unstick posts
- Delete posts
- only support hierarchical taxonomies
- WP-Cron is used
3.30.11. Backup, migrate, import, export
3.30.11.1. BackUpWordPress wp:plugin:backupwordpress
3.30.11.2. backupbuddy (iThemes) wp:plugin:backupbuddy
3.30.11.3. WP Migrate DB wp:plugin:wp-migrate-db
- With some string replace function. e.g.
//dev.bradt.ca => //bradt.ca,/sites/bradt.ca/public_html => /sites/bradt.ca/public_html - wp_options table keep track of recently_edited files that are edited on WP-admin and some other plugins might keep the filepath info
3.30.11.4. wp-migrate-db-pro, wp-migrate-db-pro-media-files
Non-free plugin. Install them in both source and destination websites.
3.30.11.5. wp-all-import-pro, wp-all-export-pro
- Basics
- License is for plugin update only
- Both plugins (WP All Import, WP All Export) do not require a license to use
- (no term)
- Create a brand new wp site, use wp cli to import .sql file and install Export-pro
- (no term)
- Export-pro to export parent posts to .csv or .xml
- (no term)
- Install Import-pro to import parent posts
- Export-pro wp:plugin:wp-all-export-pro
- Exports one post type at a time or build a WP_Query
status=publish parent=0first- Usually include all fields
- Standard
- ID
- Title
- Content
- Excerpt
- Date
- Post Type
- Permalink
Media
- Images
- wp:db:wp_postmeta:_wp_attached_file if it's an image
- URL
- wp:db:wp_options:siteurl + wp:db:wp_options:upload_path + wp:db:wp_postmeta:_wp_attached_file
- (no term)
- Filename
- (no term)
- Path
- (no term)
- ID
- (no term)
- Title
- (no term)
- Caption
- (no term)
- Description
- (no term)
- Alt Text
- (no term)
- Featured
- Attachments
- files that are other than images
- URL
- wp:db:wp_options:siteurl + wp:db:wp_options:upload_path + wp:db:wp_postmeta:_wp_attached_file
- (no term)
- Filename
- (no term)
- Path
- (no term)
- ID
- (no term)
- Title
- (no term)
- Caption
- (no term)
- Description
- (no term)
- Alt Text
- Taxonomies
- Custom
- Other
- Status
- Author ID
- Author Username
- Author Email
- Author First Name
- Author Last Name
- Slug
- Format
- Template
- Parent
- Parent Slug
- Order
- Comment Status
- Ping Status
- Post Modified Date
- Standard
- csv can split records into files while xml cannot
- Export files are saved under wp:db:wp_options:upload_path +
wpallexport/exports/hashed_codefor each export job- linux:find:newest modified files
- Import-pro wp:plugin:wp-all-import-pro
- Images
- import them to Media Library and modify the image urls inside body content after posts are imported
wp-content/uploads/wpallimport/files
- (no term)
- Taxonomy
- Taxonomy terms that don't already exist will be created
- Import file
wp-content/uploads/wpallimport/uploads/xxx/- (no term)
- Only records with the same unique_key and import id will be updated otherwise they are created
- Consider creating a custom field for unique id during the first import and use that to identify records for all imports
- Or overwrite the import file with a new one and run the previous import again
- (no term)
- Import settings > Enable
Use StreamReader instead of XMLReader to parse import file - (no term)
- Db tables
wp_pmxi_images- stores the original image urls and the corresponding attachment id
- id
- attachment_id
- e.g. image source url
- image_filename
- (no term)
wp_pmxi_posts- id
- bigint(20) unsigned
- import_id
- unique_key
- (no term)
wp_pmxi_imports- id
- d:0
- e.g. import file name
- (no term)
wp_pmxi_templates- id
- serialized exported template string
- d:''
- import template name
- Goes on
- (no term)
- For large import, choose an existing import file to creat an import using Manual Schedule, run trigger url and run processing url every 2 mins. Each processing takes about 2 minutes
- (no term)
- Custom Fields
- wpcf fields (toolset)
wpcf-custom-authorwherecustom-authoris the slug in Toolset for the custom field- WP All Import - ACF Add-on
- wp:plugin:acf
Cron
crontab -e # */2 * * * * /home/li/c/project/cs-devops/import.sh
Cron script
#!/bin/bash curl -L "https://li-dev-proj.pantheonsite.io/wp-cron.php?import_key=...&import_id=..&action=processing" -o /home/li/c/project/cs-devops/import.log
3.30.11.6. all-in-one-wp-migration wp:plugin:all-in-one-wp-migration
3.30.11.7. wordpress-importer
3.30.11.8. WP Sync DB wp:plugin:wp-sync-db
- A fork from wp:plugin:wp-migrate-db
- https://github.com/wp-sync-db/wp-sync-db
- Installation
- requires wp:plugin:github-updater
- Just download the zip
- copy the connection info from the source website and paste into the destination website in order to pull to destination
3.30.12. DB Optimize
3.30.12.1. wp-optimize
3.30.13. WP Admin
3.30.13.1. duplicate-post wp:plugin:duplicate-post
- Clone post
3.30.13.2. Admin Columns wp:plugin:codepress-admin-columns
- Free version just adds and displays columns on wp-admin
- Columns for Posts, Users, Media and Comments
- Admin Columns Pro
- https://www.admincolumns.com
- 1 year support and update. Renew at 60% of normal price
- Plans
- Personal $49 1 Site
- Business $99 5 Sites
- Developer $199 Unlimited Sites
- Sort and filter columns
- Inline field edit
- Export posts after filtering and sorting to CSV
- Define custom column sets
- Add-ons for wp:plugin:acf, wp:plugin:wooCommerce, Pods, BuddyPress, wp:plugin:wordpress-seo, wp:plugin:types, Ninja Forms, wp:plugin:the-events-calendar
3.30.13.3. tinymce-advanced
- Add, remove and arrange buttons shown on the Visual Editor toolbar.
- Enable or disable 15 plugins for TinyMCE.
- Modify options such as keeping the paragrah tags in the Text editor and importing CSS classes from the theme's editor-style.css
- Always update as wp core updates
3.30.13.4. disable-embeds wp:plugin:disable-embeds
- Disable embed, oEmbed and remove
wp-embed.min.js - https://kinsta.com/knowledgebase/disable-embeds-wordpress/
- (no term)
- Refer to wp:api:oembed
3.30.14. Cache, CDN
3.30.14.1. wp-redis wp:plugin:wp-redis
- https://wordpress.org/plugins/wp-redis/#installation
- For Pantheon, install object-cache.php to wp-content/object-cache.php with a symlink or by copying the file
- For sites not running on Pantheon, edit wp-config.php to add cache e.g. Redis credentials
- Refer to lando:service:cache
3.30.14.2. Super Cache (SuperCache)
Warning: include(/var/www/html/wp-content/plugins/wp-super-cache/wp-cache-base.php): failed to open stream
If you encounter this problem, your WP_CACHE cache folder is not set properly. Check `wp-config.php`. You should see
define('WP_CACHE',true); define('WPCACHEHOME', ABSPATH. 'wp-content/plugins/wp-super-cache/');
There's another place where the cache folder is defined: `$cache_path` in `wp-content/wp-cache-config.php`
Uninstall To manually uninstall:
• Turn off caching on the plugin settings page and clear the cache. • Deactivate the plugin on the plugins page. • Remove the WP_CACHE define from wp-config.php. It looks like define( 'WP_CACHE', true ); • Remove the Super Cache mod_rewrite rules from your .htaccess file. • Remove the files wp-content/advanced-cache.php and wp-content/wp-cache-config.php • Remove the directory wp-content/cache/ • Remove the directory wp-super-cache from your plugins directory.
If all else fails and your site is broken • Remove the WP_CACHE define from wp-config.php. It looks like define( 'WP_CACHE', true ); • Remove the rules (see above) that the plugin wrote to the .htaccess file in your root directory. • Delete the wp-super-cache folder in the plugins folder. • Optionally delete advanced-cache.php, wp-cache-config.php and the cache folder in wp-content/.
3.30.14.3. W3 Total Cache wp:plugin:w3-total-cache
- Manually remove
Files :: global search W3TC_ADDIN_FILE_
Backup and remove wp-content/plugins/w3-total-cache Rename or remove wp-content/cache Rename or remove wp-content/w3tc-config and maybe wp-content/w3tc Rename or remove Files above under wp-content root :: advanced-cache.php db.php object-cache.php and maybe wp-total-cache-config.php Remove lines related to w3tc in .htaccess Remove line from wp-config.php define('WP_CACHE', true)
- Releases
0.9.5.1 > 0.9.6 > 0.9.7
3.30.14.4. WP Rocket wp:plugin:wp-rocket
- https://wp-rocket.me by WP Media
- Plans: features are the same for all plans
- Single: 1 site
- Plus: 3 sites
- Infinite: unlimited sites
3.30.14.5. cloudflare
It includes CDN, SSL, static cotnent caching, firewall and traffic analytics tools.
On CloudFlare
- Add a website
- confirm DNS records are correct
- Change DNS namerservers to cloudflare NS
- CloudFlare is acting as a reverse proxy to your website
Initial setup https://support.cloudflare.com/hc/en-us/articles/201897700
- On web server, whitelist CloudFlare IP addresses
- Need to install mod_cloudflare Apache module in order for web server to see visitor IP
- Firewall > IP Firewall > Access Rules
- Turn off Cloudflare for a domain
Sometimes, you want to bypass Cloudflare for a domain e.g.
abc.caandwww.abc.ca.- e.g. for the first time to use Let's Encrypt (certbot) with Nginx to setup redirect https://abc.ca to https://xyz.ca and both of those websites are not hosted on Nginx
- DNS > Turn off for those domains
- Crypto > SSL set to Off
- letsencrypt:certbot
CloudFlarec.com > Crypto
- SSL set to Full or Full (strict)
- Disable
Always Use HTTPS
- TS: Restart Nginx Cause Error 525: SSL Handshake Failed
The SSL certificate on Cloudflare needs to renew
- Cloudflare dashboard for the website > Crypto > at the very bottom Disable Universal SSL
- Wait 5 minutes and turn it back on
- Still on Crypto page, under SSL > Universal SSL Status, wait until it says Active Certificate (usually it takes about 10 minutes)
Or maybe the target Nginx server's SSL certificate has expired e.g. Nginx is used to redirect this website to another website. Both websites are not hosted on Nginx
- TS: ERR_TOO_MANY_REDIRECTS
When Crypto > SSL is set to Flexible, Cloudflare sends all requests between Cloudflare and the origin web server over HTTP non-encrypted. Your Nginx receives HTTP request and might have redirect rules to redirect to https and thus causes redirect loop.
- TS: err_ssl_version_or_cipher_mismatch
This is due to the SSL given by Cloudflare is not active yet. Wait for 10 minutes.
3.30.15. Google
3.30.15.1. Google Analytics Dashboard for WP by ExactMetrics (GADWP) wp:plugin:google-analytics-dashboard-for-wp
- Show stats for each post!
- Insert GA tracking code dynamically using analytics.js
- Can add events for downloads, mailto, telephone and outbound links
- Can exclude by user roles for events
- Hook for Commands
$commands = [ // command 1 [ 'command' => 'create', 'fields' => [ 'trackingId' => 'UA-xxx', 'cookieDomain' => 'auto', ], 'fieldsobject' => [], ], // command 2 [ 'command' => 'send', 'fields' => [ 'hitType' => 'pageview', ], 'fieldsobject' => null, ] ]; // do_action( 'gadwp_analytics_commands', $this); add_action( 'gadwp_analytics_commands', function( $gadwp ) { $commands = $gadwp->get(); // var_dump($commands); $last = array_pop( $commands ); // last command always is send pageview global $post; $fields = []; $fields['option'] = 'contentGroup1'; $fields['value'] = esc_attr($post->post_type); $fieldsobject = null; $commands[] = $gadwp->prepare( 'set', $fields, $fieldsobject ); // Add a new line to the commands array // ga('set', 'contentGroup1', '<Group Name>'); $commands[] = $last; $gadwp->set($commands); // Store the new commands array });
3.30.16. SEO
3.30.16.1. Yoast SEO wordpress-seo wp:plugin:wordpress-seo
- How to SEO?
- https://yoast.com/wordpress-seo/
- API and fitlers
- https://yoast.com/wordpress/plugins/seo/api/
- Link Google Search Console
- google:search console
- (no term)
- Yoast SEO Premium: paid by number of sites
- https://yoast.com/wordpress/plugins/seo/
- Keywords, keyphrases, synonyms, related keywords and all word forms for those
- Free
- only 1 keyword or keyphrase
- Warning on most important pages, and when these are not updated for 6 months
- Get suggestions for links to other pages
- Show 5 words or phrases used the most on a page
- Redirect manager
- Focus keyword export
- Google, Facebook and Twitter previews
- Free
- only Google preview
- Both versions
- schema.org
- Flesch Reading Ease score
- Primary category
- Canonical URL
- Version
7 - UI
- Open Graph wp:plugin:wordpress-seo:og
- At least 200x200 pixels image for
og:image - Resave the post/article on WP first to prevent WP Super Cache as FB crawler gets to the page differently and then run Facebook Open Graph Object Debugger
- Refer to html:og
- required
- e.g. article, for archive page, it's
object - required
- required
{ "locale": "en_us" }- og:description
- og:site_name
- og:updated_time
<!-- Archive Page --> <meta property="og:locale" content="en_US" /> <meta property="og:type" content="object" /> <meta property="og:title" content="Archive Page Title" /> <meta property="og:url" content="https://www.a.ca/category/a-category/" /> <meta property="og:site_name" content="My Site Name" /> <meta property="og:image" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" /> <meta property="og:image:secure_url" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" /> <meta property="og:image:width" content="470" /> <meta property="og:image:height" content="351" /> <!-- Single Post Page --> <meta property="og:locale" content="en_US" /> <meta property="og:type" content="article" /> <!-- article is a namespace, all single posts use article --> <meta property="og:title" content="Post Title" /> <meta property="og:description" content="Post Excerpt By Default" /> <meta property="og:url" content="https://www.a.ca/features/a-feature-post/" /> <meta property="og:site_name" content="My Site name" /> <meta property="article:publisher" content="https://www.facebook.com/my-fb-name" /> <meta property="article:tag" content="A Tag name" /> <!-- multiple tags --> <meta property="article:section" content="Primary Category Name" /> <!-- Primary category --> <meta property="article:published_time" content="2019-04-05T19:39:44+00:00" /> <meta property="article:modified_time" content="2019-04-08T19:16:37+00:00" /> <meta property="og:image" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" /> <!-- Post Featured Image, if not, use site image --> <meta property="og:image:secure_url" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" /> <meta property="og:image:width" content="470" /> <meta property="og:image:height" content="351" />
- At least 200x200 pixels image for
- Turn off json-ld wp:plugin:wordpress-seo:json-ld
Once SEO > Search Appearance > General > Knowledge Graph > Company name and logo are specified, Plugin inserts Oranization and Website by default
{ "@context": "http:\/\/schema.org", "@type": "WebSite", "@id": "#website", "url": "https:\/\/mywebsite.com\/", "name": "My Website Name", "potentialAction": { "@type": "SearchAction", "target": "https:\/\/mywebsite.com\/?s={search_term_string}", "query-input": "required name=search_term_string" } }{ "@context": "http:\/\/schema.org", "@type": "Organization", "url": "https:\/\/mywebsite.com\/", "sameAs": [], "@id": "#organization", "name": "My Organization Name", "logo": "http:\/\/mywebsite.com\/wp-content\/uploads\/2018\/02\/logo.jpg" }Use filter to turn it off and customize it https://www.bybe.net/yoast-seo-guide-disable-schema-json-ld-wordpress/
function bybe_remove_yoast_json($data){ $data = array(); return $data; } add_filter('wpseo_json_ld_output', 'bybe_remove_yoast_json', 10, 1);
- Turn off canonical
add_filter( 'wpseo_canonical', '__return_false' ); - Setting: add exclude post from sitemap
- Edit a post
- Setting > Allow search engines to show this Post in search results
- (no term)
Theme functions.php
add_filter( 'wpseo_exclude_from_sitemap_by_post_ids', function ($post_ids) { if (function_exists('wc_get_page_id')) { $post_ids[] = wc_get_page_id( 'checkout' ); } return $post_ids; } );
- Setting: turn off Author Archives
SEO > Search Appearance > Archives > disable Author archive Otherwise /author-sitemap.xml will show the admin name
- Release
1.5.2.7, 7.1, 7.5
3.30.16.2. all-in-one-seo-pack wp:plugin:all-in-one-seo-pack
- By ServMask
3.30.17. Social media, Push Notification
3.30.17.1. WordPress to Buffer Pro
- $199 for lifetime unlimited websites or $89/yr for unlimited websites
- https://www.wpzinc.com/plugins/wordpress-to-buffer-pro/
- (no term)
- Documentaion
- (no term)
- Compatible with wp:plugin:acf, wp:plugin:the-events-calendar
- (no term)
- WP-CLI to repost
- Chrome Extension
- share any url, scan page to get images, specify which social accounts to post and add images
3.30.17.2. Simple Social Icons wp:plugin:simple-social-icons
3.30.17.3. OneSignal wp:plugin:onesignal-free-web-push-notifications
- Free to web push notifications for up to 30k subscribers and no limit on number of push notifications you can send
- Custom Code can be selected on OneSignal.com. It doesn't have to be
WordPress Plugin or Website Builder
3.30.18. Post type - Comments
3.30.18.1. Disqus
If you receive a lot of errors about dsq_sync_forum in Apache logs, it is caused by Disqus tries to sync with WP comments so frequently.
First in WP Disqus setting, check "Disable automated comment importing"
Empty all cron rows in wp-options table `UPDATE wp_options SET option_value='' WHERE option_name='cron';"`
https://wordpress.org/support/topic/high-server-load-and-dsq_sync_forum-problem?replies=6
3.30.18.2. disable-comments wp:plugin:disable-comments
- Keywords
- disable comments, remove comments
- (no term)
- Turn off comments and it requires comment to be deleted
If you want to keep comments in db, use this custom code without using the plugin.
function filter_media_comment_status( $open, $post_id ) { $post = get_post( $post_id ); if( in_array($post->post_type, array('attachment', 'post')) ) { return false; } // or simply return false; return $open; } add_filter( 'comments_open', 'filter_media_comment_status', 10 , 2 ); // other actions might be needed
Because in themes
if( comments_open() || get_comments_number() ) { // open or there's at least one comment comments_template(); } // or for custom post types if( post_type_supports( get_post_type(), 'comments' ) ) { // put comment related code here, including... if( comments_open() ) { // ...a comment form, maybe } } else { // sit back and relax }
- WP Admin > Settings > Discussion > disable
Allow People to post comments on new articlesfor future posts and disableAllow link notifications from other blogs (pingbacks and trackbacks) on new articles - Maybe useful to set
Automatically close comments on articles older than 0 days
3.30.19. Post type - Events
3.30.19.1. The Events Calendar wp:plugin:the-events-calendar
- Free
- Month View
- List View
- Day View
- Saved Content: saved Venues and Organizers
- Keyword Search
- AJAX
- iCal & Gcal Export
- Events List Widget
- Pro: https://theeventscalendar.com/product/wordpress-events-calendar-pro/ by Modern Tribe
- Recurring events
- Week View
- Photo View
- Map View
- Location Search
- Venue & Organizer View
- Advanced Widgets
- Mini Calendar Grid
- Advanced Upcoming Events list
- Featured Venue
- Countdown
- Shortcodes
- Additional Fields
3.30.19.2. all-in-one-event-calendar
- https://wordpress.org/plugins/all-in-one-event-calendar/
- Create a content type Event
[ai1ec view="monthly"]- Create a blank page and associate it with Events > Settings > Viewing Events to show events and customize view
- May create JavaScript script to embed on external websites
3.30.21. Forms, Polls, Surveys
3.30.21.1. Gravity Forms wp:plugin:gravity forms
- https://www.gravityforms.com/login/
- By Rocketgenius Inc.
- Require PHP 5.6+, MySQL 5.5+
- Can bypass license key to use the plugin
- Form setting
- Allow field to be populated dynamically
- so that populating this form field with a value using query string or hook.
- (no term)
- Deleting a field will also delete all entry data associated with that form field
- (no term)
- Input Mask
- Use a ‘9’ to indicate a numerical character
- Use a lower case ‘a’ to indicate an alphabetical character
- Use an asterick ‘*’ to indicate any alphanumeric character
- Use a question mark ‘?’ to indicate optional characters. Note: All characters after the question mark will be optional
- All other characters are literal values and will be displayed automatically
- Examples
- Date
99/99/9999- Course code
aaa 999- License Key
***-***-***
- (no term)
- Post fields enable you to create WP Post
- Shortcodes & Merge Tags
[gravityform id="1"]wp:plugin:gravity forms:sc:gravityform
[gravityform id="1" title="false" description="false"] [gravityform id=1 field_values='parameter_name1=value1¶meter_name2=value2']
- id
- required
- title
- true/false defaut:true
- description
- true/false default:true
- ajax
- true/false
- tabindex
- int
- field_values
- populate values
[gravityforms action="conditional"]
Insert message content to Admin and User Notification emails as well as the Confirmation Message that is displayed when a form is submitted
<!-- Only one condition is allowed --> [gravityforms action="conditional" merge_tag="{Number:1}" condition="greater_than" value="10"] This content would be displayed if the value of field id 1 is greater than 10. [/gravityforms] [gravityforms action="conditional" merge_tag="{Country:5}" condition="is" value="United States"] This content would be displayed if the value of field id 5 is United States. [/gravityforms] [gravityforms action="conditional" merge_tag="{Country:5}" condition="isnot" value=""] This content would be displayed if no value exists for field id 5. [/gravityforms]
- Actions
[gravityform action="*"]
Display details from user meta fields
<!-- Display details from user meta fields. Requires User Registration Add-On --> [gravityform action="user" key="ID" output="raw" /] <!-- Displays the login form. Requires User Registration Add-On --> [gravityform action="login" description="false" logged_in_message="Yay! You are logged in!" registration_link_text="Register for my super awesome site" forgot_password_text="Stop forgetting your password" /]
- Merge Tags
Populate submitted field values in notification emails, post content templates and more. https://docs.gravityforms.com/merge-tags/
- Functions
gravity_formwp:plugin:gravity_forms:f:gravity_form
- Display a form in template files
- Same as wp:plugin:gravity forms:sc:gravityform
Refer to wp:plugin:gravity forms:filter:gform_field_value_$parameter_name
gravity_form( $id_or_title, $display_title = true, $display_description = true, $display_inactive = false, $field_values = null, $ajax = false, $tabindex, $echo = true );
gravity_form_enqueue_scripts
- Placement
- before
wp_head()is called
gravity_form_enqueue_scripts( $form_id, $is_ajax = false);
- Hooks and actions
gform_field_value_$parameter_namewp:plugin:gravity forms:filter:gform_field_value_$parameter_name
Populate any field with parameter
your_parameterwith the result of the functionmy_custom_population_functionadd_filter( 'gform_field_value_your_parameter', 'my_custom_population_function' ); function my_custom_population_function( $value, $field, $name ) { return 'boom!'; // populate from a cookie return $_COOKIE['utm_campaign']; // populate a list field // new way $list_array = array( array( 'Column 1' => 'row1col1', 'Column 2' => 'row1col2', 'Column 3' => 'row1col3', ), array( 'Column 1' => 'row2col1', 'Column 2' => 'row2col2', 'Column 3' => 'row2col3' ), ); return $list_array; // old way return array( 'row 1 - col1', 'row 1 - col2', 'row 1 - col3', 'row 2 - col1', 'row 2 - col2', 'row 2 - col3', 'row 3 - col1', 'row 3 - col2', 'row 3 - col3' ); } // Populate multiple fields using one function add_filter( 'gform_field_value', 'populate_fields', 10, 3 ); function populate_fields( $value, $field, $name ) { $values = array( 'field_one' => 'value one', 'field_two' => 'value two', 'field_three' => 'value three', ); return isset( $values[ $name ] ) ? $values[ $name ] : $value; } // Populate a time field add_filter( 'gform_field_value_time', 'populate_time' ); function populate_time( $value ) { $local_timestamp = GFCommon::get_local_timestamp( time() ); return date_i18n( 'h:i A', $local_timestamp, true ); // Populate a fixed date return '10/10/2010'; } // Populate author email add_filter( 'gform_field_value_author_email', 'populate_post_author_email' ); function populate_post_author_email( $value ) { global $post; $author_email = get_the_author_meta( 'email', $post->post_author ); return $author_email; }
gform_field_containergravityforms:gform_field_container
add_filter( 'gform_field_container', 'add_bootstrap_container_class', 10, 6 ); function add_bootstrap_container_class( $field_container, $field, $form, $css_class, $style, $field_content ) { $id = $field->id; $field_id = is_admin() || empty( $form ) ? "field_{$id}" : 'field_' . $form['id'] . "_$id"; return '<li id="' . $field_id . '" class="' . $css_class . ' form-group">{FIELD_CONTENT}</li>'; }
- Query String
Populate form fields
http://siteurl.com/form-url/?your_parameter=value - Events
- CSS Styling
https://docs.gravityforms.com/basic-structure/ https://docs.gravityforms.com/css-targeting-examples/ https://docs.gravityforms.com/css-visual-guide/ https://docs.gravityforms.com/css-ready-classes/
<div class="gform_body"> <ul id="gform_fileds_1" class="gform-fields top_label form_sublabel_below field_description_below"> <!-- field #1 --> <li id="field_1_7" class="gfield field_sublabel_below field_description_below gfield_visibility_visible"> <label for="input_1_7" class="gfield_label">Label 1</label> <div class="ginput_container ginput_container_text"> <input type="text" name="input_7" id="input_1_7" value class="medium" tabindex="1" aria-invalid="false"> </div> </li> <!-- field #2 --> </ul> </div>
- Use gravityforms:gform_field_container to modify field container
<li> - Use SASS to assign Bootstrap classes. Refer to sass:@extend
- Use gravityforms:gform_field_container to modify field container
- Add-Ons
- WP Config
https://docs.gravityforms.com/wp-config-options/
define( 'GF_LICENSE_KEY', 'YOUR-LICENSE-KEY-HERE' );
- Database
- wp_gf_form
- forms
- TS:
jQuery is not found
Gravity Forms by default will write an inline JavaScript call to jQuery on every form you add to a page. This will throw an error if you’re loading jQuery in the footer of your site (which you should be doing).
add_filter("gform_init_scripts_footer", "init_scripts"); function init_scripts() { return true; }
3.30.21.2. Gravity Perks
- https://gravitywiz.com/documentation/ By Gravity Wiz
3.30.21.3. Crowdsignal wp:plugin:crowdsignal
- https://crowdsignal.com by Automattic. Formerly Polldaddy
Plan
- Pro: $17/month or $200/yr
- Unlimited polls and surveys
- 1 user account
- No Crowdsignal branding
- Can't customize domain
- 10k email/month invites
- Sharing
- Permalink
- iframe, HTML/JavaScript, WordPress shortcode (WP Crowdsignal plugin), QR Code
- Send emails (in Email Groups) to invite them to take a survey
- Receive GDPR erasure requests from receivers who want to delete their responses and votes
- Email Group
- Every member/contact in any email group has 4 fields: email address, first name, last name and custom data
- ask one simple question
3.30.21.4. contact-form-7
- Shortcode
Add html id
contact-form-7 id="1234" title="Contact form 1" html_id="contact-form-1234" html_class="form contact-form"]
- autop wp:p:contact-form-7:autop
wp-config.php
define( 'WPCF7_AUTOP', false );
- hooks - reCaptcha
Default template
[response] <p><label for="your-name">Your Name (required)</label><br> [text* your-name id:your-name]</p> <p><label for="your-email">Your Email (required)</label><br> [email* your-email id:your-email]</p> <p><label for="your-message">Your Message</label><br> [textarea* your-message id:your-message]</p> <p>[submit "Send"]</p>
Turn off wp:p:contact-form-7:autop
- filter:wpcf7_form_elements
Remove <noscript> for reCaptcha which has an iframe that is not WCAG ready
function csWpcf7FormElements($form) { try { $dom = new DOMDocument; $dom->loadHTML($form); $noscripts = $dom->getElementsByTagName('noscript'); while ($noscripts->length > 0) { $i = $noscripts->item(0); $i->parentNode->removeChild($i); } return $dom->saveHTML(); } catch (Exception $err) { return $form; } return $form; } add_filter( 'wpcf7_form_elements', 'csWpcf7FormElements');
- filter:shortcode_atts_wpcf7 wp:p:contact-form-7:f:shortcode_atts_wpcf7
add_filter( 'shortcode_atts_wpcf7', 'custom_shortcode_atts_wpcf7_filter', 10, 3 ); function custom_shortcode_atts_wpcf7_filter( $out, $pairs, $atts ) { $my_attr = 'destination-email'; if ( isset( $atts[$my_attr] ) ) { $out[$my_attr] = $atts[$my_attr]; } return $out; }
- action:wpcf7_before_send_mail wp:p:contact-form-7:a:wpcf7_before_send_mail
- filter:wpcf7_form_elements
- DOM Events
https://contactform7.com/dom-events/
wpcf7invalid — Fires when an Ajax form submission has completed successfully, but mail hasn’t been sent because there are fields with invalid input. wpcf7spam — Fires when an Ajax form submission has completed successfully, but mail hasn’t been sent because a possible spam activity has been detected. wpcf7mailsent — Fires when an Ajax form submission has completed successfully, and mail has been sent. wpcf7mailfailed — Fires when an Ajax form submission has completed successfully, but it has failed in sending mail. wpcf7submit — Fires when an Ajax form submission has completed successfully, regardless of other incidents.
event.detail detail.inputs detail.contactFormId The ID of the contact form. detail.pluginVersion The version of Contact Form 7 plugin. detail.contactFormLocale The locale code of the contact form. detail.unitTag The unit-tag of the contact form. detail.containerPostId The ID of the post that the contact form is placed in.
Example redirect to a page
document.addEventListener( 'wpcf7mailsent', function( event ) { var inputs = event.detail.inputs; if ( '123' == event.detail.contactFormId) { location = 'http://example.com/'; } }, false ); document.addEventListener( 'wpcf7submit', function( event ) { var inputs = event.detail.inputs; for ( var i = 0; i < inputs.length; i++ ) { if ( 'your-name' == inputs[i].name ) { alert( inputs[i].value ); break; } } }, false );
- Form tags, akismet
Select Country (required) [checkbox* your-country use_label_element "China" "India" "San Marino"] Select Sports [radio your-sports label_first default:2 "Football" "Tennis" "Pole-vault"] Select Fruit (required) [checkbox* your-fruit exclusive "Apple" "Banana" "Grape"] Select Browser (required) [select* your-browser include_blank "Firefox" "Safari" "Opera" "Lynx"] Select Ghkdsjdf [select your-ghkdsjdf multiple "fsdfs" "klgjfk" "vdrie"] [submit "Send"]
- Akismet
akismet:author Add this option to the field that accepts the name of the sender. Example: [text* your-name akismet:author] akismet:author_email Add this option to the field that accepts the email address of the sender. Example: [email* your-email akismet:author_email] akismet:author_url Add this option to the field that accepts the URL of the sender. Example: [text your-url akismet:author_url]
Test viagra-test-123 as name
- File upload
https://contactform7.com/file-uploading-and-attachment/
[file your-file filetypes:pdf|txt limit:2mb] //id:foo //limit:1048576 //limit:1024kb //limit:1mb // multiple classes // class:y2008 class:m01 class:d01
Default file upload location is wp-content/uploads/wpcf7_uploads
- select with pipes
[select your-recipient "CEO|ceo@example.com" "Sales|sales@example.com" "Support|support@example.com"]Only the part before pipes are revealed on the front end to prevent spam. Use [your-recipient] in the Mail template. To get value before pipe, [_rawfield name] e.g. [_raw_your-recipient]
- Attribute - default
// The field will obtain its default value from the GET variable with the same name (“your-name”) // http://example.com/contact/?your-name=John+Smith // John Smith [text* your-name default:get] // This form-tag has two default options and a value “Your Name”. The options are evaluated from the first to the last. In this example, default:get is evaluated first. If the “your-name” GET variable has a value, it will be used for the default value. If that value is empty, default:post_meta will be evaluated next. If both of these options do not have values, “Your Name” will be used. [text* your-name default:get default:post_meta "Your Name"]
Your Name: [text* your-name default:user_display_name] Your E-mail: [email* your-email default:user_email]
Get default from shortcode attributes
[contact-form-7 id="123" title="Contact Form" destination-email="xxxxxx@example.com"] [email* destination-email default:shortcode_attr] // Filter needs to setup to catch custom attributes. Refer to wp:p:contact-form-7:f:shortcode_atts_wpcf7
- hidden fields
id:foo,[hidden your-text class:y2008 class:m01 class:d01] [hidden your-email default:user_email "example@example.com"]
- Akismet
- multifile-upload-field-for-contact-form-7
Form tag [multifile your-photos] to upload multiple files as one zip file
- contact-form-7-confirm-email-feild
Confirm email field
[email* your-email akismet:author_email] <label> <p>Confirm Email:</p> [confirm_email* confirm-email]</label>
- flamingo
save form submission By default, flamingo will take [your-subject] [your-name] [your-email]. Overwrite them
Additional Settings in the contact form
flamingo_email: "[the-email-field]" flamingo_name: "[the-name-field]" flamingo_subject: "[the-subject-field]"
- contact-form-plugin
receive messages from customers to your email addresses. Download, activate and paste [bestwebsoft_contact_form] shortcode on any page, post or widget to display the form. Customize the form styles and contents easily with the pre-build options.
- Release
3.7.2 4.0 :: WP 3.9+ 4.1 :: WP 4.0+ 4.9 :: WP 4.7+ 5.0 :: WP 4.8+, on_sent_ok and on_submit are removed 5.0.1
3.30.21.5. constant-contact-forms
Require WP 4.0+ and PHP 5.4+ Widget: Constant Contact Form shortcode: [ctct form="5872"]
- Setting
Enable logging
- wp filters
Add label on ConstantContact.com for custom field values add_filter( 'constant_contact_include_custom_field_label', '__return_true' );
- Caveats
There's no name field only first name and last name field captured on ConstantContact.com
If there're more than one Constant Contact forms, the form fields will have the same id. This will create a problem when clicking a later opt-in checkbox label always jump to the first opt-in checkbox
Make opt-in checkbox required and fix opt-in checkbox label
$(function(){ $('label[for=ctct-opt-in]').prev().addClass('ctct-form-field-required').prop('required',true);; $('label[for=ctct-opt-in]').on('click', function(e){ e.preventDefault(); e.stopPropagation(); var $input = $( this ).prev(); // previous sibling $input.prop('checked', !$input.prop("checked")); // jQuery 1.6+ }); });
3.30.21.6. constant-contact-forms-by-mailmunch
It takes way too much work to customize the form..
3.30.22. Site search
3.30.22.1. WP Advanced Search wp:plugin:wpas
https://github.com/raideus/wp-advanced-search http://wpadvancedsearch.com/docs/setup/
- Copy code to wp-content/mu-plugins/wp-advanced-search
- wp-content/mu-plugins/load-wpas.php
define( 'WPAS_URI', WPMU_PLUGIN_URL . '/wp-advanced-search' ); require_once( 'wp-advanced-search/wpas.php' ); // define your forms function my_search_form() { $args = array(); $args['wp_query'] = array('post_type' => 'post', 'posts_per_page' => 5); $args['fields'][] = array('type' => 'search', 'title' => 'Search', 'placeholder' => 'Enter search terms...'); $args['fields'][] = array('type' => 'taxonomy', 'taxonomy' => 'category', 'format' => 'select'); register_wpas_form('my-form', $args); } add_action('init', 'my_search_form');
- Fields
http://wpadvancedsearch.com/docs/template-setup/appearance/
- Works well with js:lib:select2
// pre_html and post_html $args['fields'][] = array( 'type' => 'search', 'placeholder' => 'Enter search terms...', 'pre_html' => '<div class="row">', 'post_html' => '</div>' ); // class $args['fields'][] = array( 'type' => 'search', 'placeholder' => 'Enter search terms...', 'class' => array( 'myclass', 'anotherclass' ) ); // use an html-type field to display messages $args['fields'][] = array( 'type' => 'html', 'value' => '<div class="my-element"><h3>Some custom HTML content here</h3></div>' ); // disable wrapper for the whole form $args['form'] = array( 'disable_wrappers' => true ); // works well with js:lib:select2 $args['fields'][] = array( 'type' => 'html', 'value' => '<div class="form-row">', ); $args['fields'][] = array( 'type' => 'meta_key', 'meta_key' => 'wpcf-listing-city', 'placeholder' => 'City...', 'format' => 'text', 'class' => array( 'col-auto', 'form-control form-control-lg' ), 'pre_html' => '<div class="form-group d-none d-md-block col-md-4">', 'post_html' => '</div>', 'allow_null' => 'ALL', 'operator' => 'AND', 'data_type' => 'CHAR', 'compare' => 'LIKE', ); // meta_key with select dropdown $query = " SELECT DISTINCT CAST(pm.meta_value AS DATE) FROM wp_posts p INNER JOIN wp_postmeta pm ON pm.post_id = p.id WHERE p.post_type = 'agenda' AND p.post_status = 'publish' AND pm.meta_key = 'agenda_date' ORDER BY CAST(pm.meta_value AS DATE) "; global $wpdb; $agenda_date = $wpdb->get_col($query); $agenda_date_form = array(); foreach($agenda_date as $d) { $agenda_date_form[$d] =$d; } $args['fields'][] = array( 'type' => 'meta_key', 'meta_key' => 'agenda_date', 'format' => 'select', 'data_type' => 'DATE', 'values' => $agenda_date_form, 'allow_null' => 'Select a date', // treat the first key/value pair in values as null 'class' => array( 'col-auto', 'form-control form-control-lg' ), 'pre_html' => '<div class="form-group col-md-6">', 'post_html' => '</div>', ); $args['fields'][] = array( 'type' => 'taxonomy', 'taxonomy' => 'listing_category', 'format' => 'multi-select', 'class' => array( 'col-auto', 'select2 form-control form-control-lg' ), 'pre_html' => '<div class="form-group d-none d-md-block col-md-9">', 'post_html' => '</div>', 'allow_null' => 'ALL', 'nasted' => true, 'operator' => 'IN' ); $args['fields'][] = array( 'type' => 'html', 'value' => '</div>', );
$args['wp_query']- Other configuration
- Form configuration
$args['form']
http://wpadvancedsearch.com/docs/config/form-config/
$args['form'] = array( // arguments here );
AJAX form http://wpadvancedsearch.com/docs/config/form-config/ajax/
$args['form']['ajax'] = array( // arguments here );
- Display forms
$my_search = new WP_Advanced_Search('my-form'); // display form // ob_start(); $my_search->the_form(); // $form_output = ob_get_clean(); // $form_output = sprintf( '<div class="row"><div class="col p-4">%s</div></div>', $form_output); // echo $form_output // display search result $query = $my_search->query(); // you can also modify the global $wp_query // global $wp_query; // $temp = $wp_query; // $wp_query = $search->query(); // run main loop e.g. if (have_posts()) { ... } // wp_reset_query(); // $wp_query = $temp; // this might not be necessary if ( $query->have_posts() ): while ( $query->have_posts() ): $query->the_post(); the_title(); endwhile; endif; // pagination :: use the same arguments wp core pageinate_links() // https://codex.wordpress.org/Function_Reference/paginate_links // Refer to wp:theme:pagination $my_search->pagination(array('prev_text' => '«','next_text' => '»')); // result count $args = array(); echo 'Displaying results ' . $my_search->results_range($args) . ' of ' . $wp_query->found_posts; //pre – Content to display before the output //marker – Content to use as the range marker (ie passing a hyphen “-” would show the range as “1-5″) //post – Content to display after the output $args = array('pre' => '', 'marker' => '-', 'post' => '');
- Alter query with pre_get_post
- Use wp:action:pre_get_posts
- orderby/order provided by this plugin is not strong. Need to alter it
add_action( 'pre_get_posts', 'lili_pre_get_posts' ); function lili_pre_get_posts( $query ) { if (! is_admin()) { if ( $query->is_main_query() ) { // WPAS is not a main query } else { // not main query // Agenda Archive display with WPAS if ( in_array( $query->get( 'post_type' ), array( 'agenda' ) ) ) { // $query->set( 'posts_per_page', '-1' ); // this was set in WPAS $args['wp_query']['posts_per_page'] = -1 // need to add clauses in order to sort custom fields $meta_query = array( 'date_clause' => array( 'key' => 'agenda_date', 'compare' => 'EXISTS', ), 'start_clause' => array( 'key' => 'agenda_start_time', 'compare' => 'EXISTS', ), ); // merge WPAS meta query $old_meta_query = $query->get('meta_query'); $old_meta_query = (empty($old_meta_query)) ? array() : $old_meta_query; $meta_query = array_merge($meta_query, $old_meta_query); $query->set( 'meta_query', $meta_query ); $query->set( 'orderby', array( 'date_clause' => 'ASC', 'start_clause' => 'ASC', 'date' => 'ASC', ) ); } } } return $query; }
3.30.22.2. custom-search-plugin
Add custom post types and taxonomies to WordPress website search results. A quick and easy way to search everything within custom post types and taxonomies.
3.30.23. Redirect, Permalinks
3.30.23.1. custom-permalinks
Change URL to any structure for individual post, page, tag or category. Need to resave Permalinks setting
3.30.23.2. redirection wp:plugin:redirection
- https://wordpress.org/plugins/redirection/
- Redirect without .htaccess
- Support regex, ignore slash (simple text source url), ignore case
- Monitor posts including CPT permalink changes and create redirect
3.30.24. Security
3.30.24.1. hc-custom-wp-admin-url
3.30.24.2. better-wp-security (iThemes Security)
https://en-ca.wordpress.org/plugins/better-wp-security/
Latest plugin version usually requires the latest WP Core. PHP version 5.2+ Site has to run either on Apache, LightSpeed or Nginx Work on multi-site
Will add lines to .htaccess and wp-config.php
Banned Users :: get blacklisted IPs, user agents and apply to your website. Enable HackRepair.com's blacklist feature
- HackRepair is enalbed, then .htaccess is modified to add user agents and referrer rules for blocking (Forbidden) access.
Local Brute Force Protection :: Ban hosts and users with invalid login attempts 404 Detection :: locks out someone that gets too many 404 pages Global Settings :: Lockout White List by IP
Network Brute Force Protection :: enabled by default WordPress Tweaks : enabled by default
Away mode :: Disable access to wp Dashboard on a schedule
Scan site to report vulnerabilities and try to fix them Ban bad user agents, bots and other hosts Enforce strong passwords for all accounts Force SSL for admin pages and any pages Turn off file editing from wp-admin UI Detect and block attacks to filesystem and database Change URLs of login, admin Turn off the ability to login for a given time period Remove theme, plugin and core update notifications from users who do not have permission to update them Remove Windows Live Write header info Remove RSD header info Rename admin account Change ID on the user with ID 1 Change database table prefix Change wp-content path Remove login error messages
3.30.24.3. exploit-scanner wp:plugin:exploit-scanner
Search the files on your website, and the posts and comments tables of your database for anything suspicious. It also examines your list of active plugins for unusual filenames.
You need to have wp-content/plugins/exployt-scanner/hashes-4.9.6.php to match the wp version in order for the file checking to work correct https://github.com/philipjohn/exploit-scanner-hashes
3.30.25. Authentication
3.30.25.1. WordPress OAuth Server
- Premium without trial https://wp-oauth.com
- Installation
- Plugin > Settings > General
- OAuth Server Enabled
- Plugin > Settings > Advanced Config
- Authorization Code
- HTTP redirects and WP login form when authenticating
- Allow Implicit
- Enable
Plugin > create a client
- Redirect URI
- Get the client ID
- Open a InCognito Window and test the Auth
http://mylocalwpauthserver:8000/oauth/authorize?client_id={xxx}&response_type=tokenit will redirect to login page, after login, it will redirect to the Redirect URI previously set- Get the Bearer token from the Redirect URI
https://redirected.io/#access_token={xxx}&expires_in={yyy}&token_type=Bearer&other
- Plugin > Settings > General
3.30.26. Email
3.30.26.1. sendgrid-email-delivery-simplified
- Create API key with
Mail Send: Fullpermission only - Create Sender Authentication: Domain and Link Branding
- On WP, use
Send Mail with API - Specify category
yourwebsite.com - Hard set API key in code otherwise the setting page keeps refreshing as browser form auto-fill kicks in.
wp-config.php
define('SENDGRID_API_KEY', 'your.api.key');
3.30.27. Menu, Nav, Breadcrumb
3.30.27.1. menu-image
3.30.27.2. megamenu megamenu-pro wp:plugin:megamenu
- Install free version first
- https://wordpress.org/plugins/megamenu/ then install megamenu-pro
https://www.megamenu.com/documentation/
To enable Max Mega Menu for a menu, put this menu into a theme location.
- Search box
Create a menu with custom link, go to Mega Menu > Replacements > Type Search box
- Icon color
- Don't need to modify Styling, everything is under Replacements
- CSS
- Desktop menu
/* div#mega-menu-wrap-primary */ #{$wrap} { /* ul#mega-menu-primary */ #{$menu} { /* Top level menu items for desktop and mobile */ & > li.mega-menu-item { & > a.mega-menu-link { } /* sub menu of the Top level menu */ & > ul.mega-sub-menu { } } } }
- Mobile menu
/** Push menu onto new line **/ #{$wrap} { clear: both; /* .mega-toggle-* mobile menu */ /* Evenly distributed on the first column */ .mega-toggle-blocks-left { .mega-toggle-block { width:25%; text-align:center; margin-left:0!important; a.mega-icon { &:before { display:block; } &:hover, &:focus { &:before { color:#cd1713; } } } } } /* Hide middle column */ .mega-toggle-blocks-center { display:none!important; } }
- Desktop menu
- Turn off mobile menu
Set breakpoint to 0
- WPML
WPML > Languages > Make themes work multilingual > enable Adjust IDs for multilingual functionality
Create a menu for each language using WPML
However, menus of the same theme location can only have one Mega Menu's
Menu Theme. e.g. add translation to Mobile Menu under Menu Theme[wpml_if lang='en']MENU[/wpml_if][wpml_if lang='de']Menü[/wpml_if]- It's also possible to translate the logo URL
[wpml_if lang='en']http://www.google.com[/wpml_if][wpml_if lang='de']http://www.google.de[/wpml_if]
- TS: Menu height zero
Usually it's caused by the first menu item is set to align left and the other items set to align right.
Set the default alignment in Mega Menu > Menu Themes > Menu Bar > Menu Items Align to left and set the menu item to Default in Appearance > Menus > Menu Structure > Mega Menu > Settings > Menu Item Align > Default (change from align left)
3.30.27.3. breadcrumb-navxt
After installing and activating the plugin, to get breadcrumb trails to display either use the included widget, or call the breadcrumb trail in your theme (or child theme). See the Calling the Breadcrumb Trail article for more information on calling the breadcrumb trail.
3.30.28. Slider, Lightbox
3.30.28.1. responsive-lightbox
https://dfactory.eu/docs/responsive-lightbox/manual-usage/ By default, RL applies effects to all published images, videos and galleries.
Settings > Responsive Lightbox > General settings
- set Selector and enable lightbox for WordPress image links. This is the manual way.
- (default) Enable lightbox for WordPress image links will apply to all published images
- set Display single post images as a gallery. This is the manual way to create gallery
- (default) Add lightbox to WordPress image galleries by default.
- (default) Add lightbox to YouTube and Vimeo video links by default.
<!-- Manual on single image --> <a href="image-link" data-rel="lightbox"><img class ="some-text-here" src="image-link" /></a> <!-- Manual on single gallery --> <a href="image-link" data-rel="lightbox-gallery-182"><img class ="some-text-here" src="image-link" /></a>
3.30.28.2. Slider Revolution, revslider wp:plugin:revslider
- On Envato
Display in template
putRevSlider("home"); putRevSlider("home","homepage"); // only display if current page is homepage putRevSlider("home","2,10"); // only display if current page has an ID
- Shortcode:
[rev_slider home] - Add Revolution Slider widget to the desired sidebar
3.30.28.3. LayerSlider Responsive WordPress Slider Plugin wp:plugin:LayerSlider
3.30.28.4. MetaSlider wp:plugin:metaslider
- https://www.metaslider.com by Updraft
3.30.29. Widgets
3.30.29.1. widget-logic
3.30.29.2. Mini Loops wp:plugin:mini-loops
- Old
- Miniloops
- (no term)
- Used inside shortcode
[miniloop]image, ml_imageattributes
[image from="thumb"]- get featured image.
fromcan also be and comma-separated
- attached
- attached image
- customfield
- (no term)
- first
[image class="myclass"][image width="100"][image height="100"- ">0 to scale
[image fallback="/wp-content/upload/a.jpg"]- ">clear to not save to db and always crop/scale image on the fly
3.30.29.3. vertical-marquee-plugin
In your WordPress administrator section go to Settings menu and select Vertical marquee plugin menu to configure this plugin.
Scroll Amount : This is used to organize the speed of the scrolling, only integer allowed and higher value provides a faster speed Scroll Delay : This is used to reduce the speed of the scrolling, only integer allowed.
Drag and drop the widget : Go to widget menu and drag and drop the Vertical marquee widget to your sidebar location.
Short code for pages and posts : Use the below short code in the pages and posts.
[vertical-marguee setting="1" group="group1"]
Add directly in the theme : Add the below PHP code in your theme PHP file, for example if you want to add this slider in your website footer, just activate the plugin and add this code in footer.php file.
From version 1.0 to 4.0
<?php verticalmarquee(); ?>
From version 4.0 onwards
<?php vmarquee( $setting="1", $group="widget" ); ?>
3.30.29.4. vertical-scroll-recent-post
3.30.29.5. black-studio-tinymce-widget
- Add a "Visual Editor" widget like WYSIWYG TinyMCE switching between Visual and HTML mode
- Since WP Core 4.8, it has Visual text widget but it's minimal
3.30.29.6. recent-posts-widget-extended
3.30.29.7. widget-css-classes
3.30.30. Ecommerce
3.30.30.1. WP Job Manager wp:plugin:wp-job-manager
- Basic
- Core plugin is free but add-ons are not
- New user role
employer - single.php or single-job_listing.php
- content-widget-job_listing.php
- Shortcodes
- submit_job_form
- new page out of wp-admin for employers to post new jobs
- job_dashboard
- new page out of wp-admin for employers to manager job listings
- jobs
- new page for visitors to browse, search and filter job listings
- Setting
- General
- Enable Usage Tracking
- General
- Add-ons
Add-on bundle includes several add-ons for a cheaper price
- $125/yr for one site and $250/yr for unlimited sites. It includes
- Resume Manager
- Job Alerts
- Indeed Integration
- Job Tags
- WC Paid Listings
- Simple Paid Listings
- Application Deadline
- Bookmarks
- Applications
- Embeddable Job Widget
- ZipRecruiter
- WC Paid Listings
- Require wp:plugin:woocommerce
3.30.30.2. Easy Digital Downloads wp:plugin:easy-digital-downloads
- https://easydigitaldownloads.com by Sandhills Development
- Sell digital products: eBooks, WP plugins, PDF etc.
3.30.30.3. woocommerce wp:plugin:woocommerce
- Basics
- https://docs.woocommerce.com/documentation/plugins/woocommerce/getting-started/
- https://github.com/woocommerce/woocommerce
- https://docs.woocommerce.com/documentation/plugins/woocommerce/woocommerce-codex/
- WooCommerce > Extensions > WooCommerce.com Subscriptions
- https://github.com/woocommerce/woocommerce/wiki/Database-Description
- Please note, each table name will be prefixed with your WP Database Table Prefix e.g.
wp_ - The database scheme is defined in
woocommerce/includes/class-wc-install.php
- Please note, each table name will be prefixed with your WP Database Table Prefix e.g.
- https://docs.woocommerce.com/document/installed-taxonomies-post-types/
- It creates some pages
- To custom page for Cart and Checkout
- WooCommerce > Settings > Checkout
- To check whether they are installed or to install them
- WooCommerce > System Status > Tools
- For My Account
- WooCommerce > Settings > Accounts
- For Shop page (archive page for all products)
- WooCommerce > Settings > Products
function wc_*- Starting v2.1 shortcodes are replaced with endpoints
- My Account page endpoints
- WooCommerce > Settings > Accounts
- e.g. yoursite.com/my-account/edit-account
- Checkout page endpoints
- WooCommerce > Settings > Checkout
- retrieve URL using
$order$order->get_checkout_payment_url( $on_checkout = false );$order->get_checkout_order_received_url();
- retrieve URL using
- Payment Gateways
- https://woocommerce.com/posts/woocommerce-payments-affordably/
- Stripe and PayPal Powered by Braintree. Old WooCommerce plugin can also download these gateways for free
- hooks
- Add a custom field to Order :: woocommerce_shop_order_search_fields, manage_shop_order_posts_custom_column
Add search on custom field
add_filter( 'woocommerce_shop_order_search_fields', 'lili_wc_shop_order_search_field_myfield' ); function lili_wc_shop_order_search_field_myfield( $search_fields ) { $search_fields[] = '_myfield'; // You can add a WooCommerce builtin order field without creating a field // start with _ return $search_fields; }
Add a column to the 3rd position
function ra_order_tax_receipt_columns($columns) { $columnsF = array_slice($columns, 0, 2); $columnL = array_slice($columns, 2); return array_merge($columnsF, array('myfield' => __('My Field Name')), $columnL); } add_filter('manage_edit-shop_order_columns' , 'ra_order_tax_receipt_columns', 2); add_action( 'manage_shop_order_posts_custom_column' , 'lili_wc_shop_order_search_field_myfield_content', 10, 2 ); function lili_wc_shop_order_search_field_myfield_content( $column ) { global $post, $woocommerce, $the_order; $order_id = $the_order->id; switch ( $column ) { case 'myfield' : $order_items = $the_order->get_items(); $myVarOne = get_post_meta($order_id, '_myfield', true ); echo $myVarOne; break; } }
- Page/Template hooks
- Add a custom field to Order :: woocommerce_shop_order_search_fields, manage_shop_order_posts_custom_column
- Product
$_pf = new WC_Product_Factory(); $_p = $_pf->get_product( 123 ); echo apply_filters( 'woocommerce_loop_add_to_cart_link', sprintf( '<a href="%s" rel="nofollow" data-product_id="%s" data-product_sku="%s" class="button product_type_simple add_to_cart_button ajax_add_to_cart">%s %s</a>', esc_url( $_p->add_to_cart_url() ), esc_attr( $_p->get_id() ), esc_attr( $_p->get_sku() ), implode( ' ', array_filter( [ 'button', 'product_type_' . $_p->product_type, $_p->is_purchasable() && $_p->is_in_stock() ? 'add_to_cart_button' : '', $_p->supports( 'ajax_add_to_cart' ) ? 'ajax_add_to_cart' : '', ] ) ), esc_attr( $_p->product_type ), $_p->get_price_html(), esc_attr( isset( $class ) ? $class : 'button' ), esc_html( $_p->add_to_cart_text() ) ), $_p );
For each product edit page, change Advanced > Menu order and then use menu_order in query:
$args = array( 'post_type' => 'product', 'product_cat' => $prodCat, "orderby"=>"menu_order", "order"=>"ASC"); $loop = new WP_Query( $args );
- Template Structure
https://docs.woocommerce.com/document/template-structure/
wp-content/plugins/woocommerce/templates/emails/admin-new-order.php=>wp-content/themes/yourtheme/woocommerce/emails/admin-new-order.phpwp-content/plugins/woocommerce/templates/content-single-product.php=>wp-content/themes/yourtheme/woocommerce/content-single-product.php - Test Credit Cards
Turn on test mode for each Payment Gateway. For 38.39.2, WooCommerce > Settings > Payments > Moneris - Credit Card > Manage > Environment > Sandbox
- Uninstall and remove data
In order to remove data in db when removing the plugin, add this first then deactivate and delete the plugin:
define( 'WC_REMOVE_ALL_DATA', true); /* That's all, stop editing! Happy blogging. */
- extension: Subscription
Make recurring payment and subscription products possible. Docs: https://docs.woocommerce.com/document/subscriptions/ https://woocommerce.com/products/woocommerce-subscriptions/
- extension: Name Your Price
Set custom price by buyer. Compatible with Subscription extension.
- extension: WooCommerce Sequential Order Numbers Pro
The post id of order is still not sequential but global $the_order->id is.
3.30.30.4. woo-order-export-lite
3.30.30.5. woocommerce-pdf-invoices-packing-slips
- Template hooks
http://docs.wpovernight.com/woocommerce-pdf-invoices-packing-slips/pdf-template-action-hooks/ http://hookr.io/plugins/woocommerce-pdf-invoices-packing-slips/
Simple template wp-content/plugins/woocommerce-pdf-invoices-packing-slips/templates/Simple/ html-document-wrapper.php invoice.php
- action wpo_wcpdf_before_document
$wpo_wcpdf_export_template_type :: The wpo wcpdf export template type. $wpo_wcpdf_export_order :: The wpo wcpdf export order.
function action_wpo_wcpdf_before_document( $wpo_wcpdf_export_template_type, $wpo_wcpdf_export_order ) { // make action magic happen here... }; // add the action add_action( 'wpo_wcpdf_before_document', 'action_wpo_wcpdf_before_document', 10, 2 );
- action wpo_wcpdf_after_document_label
- action wpo_wcpdf_before_billing_address
- action wpo_wcpdf_before_document
3.30.31. Theme
3.30.31.1. wpfront-scroll-top
3.30.32. Page Builder
3.30.32.1. Elementor
3.30.32.2. Beaver Builder
3.30.32.3. Divi Builder
3.30.32.4. Thrive Architect
3.30.32.5. SiteOrigin Page Builder
- https://siteorigin.com/page-builder/
- Free to start
- 20 themes that support Page Builder plugin (separate install)
- 16 SiteOrigin Site Packs: separate install, each is a full website
- Can work with free version of the following plugins
- Widgets Bundle
- about 23 widgets
- (no term)
- SiteOrigin CSS
- (no term)
- SiteOrigin Installer
- Premium version
- Addons
- Enhanced Page Builder
- Extra widgets in plugin Widgets Bundle
- Theme Enhancements
3.30.32.6. WPBakery (old name: Visual Composer)
Config
# BEGIN WordPress # ... # END WordPress SubstituteMaxLineLength 10m # if still doesn't work #WPBakery Timeout fixer - .htaccess SubstituteMaxLineLength 10M LimitRequestBody 9999999
3.30.33. jetpack
- Build wp with themes
- unlimited image CDN from Photon
- lazy loading images
- brute force attack protection
- downtime monitoring
- automated social media posting
- $3.5, $9, $29/month
3.31. Debug, WP_Error
https://codex.wordpress.org/Function_Reference/WP_Error Function may return WP_Error object. Catch error
$post = wp_insert_post([]); if ( is_wp_error($post) ) { echo $post->get_error_message(); }
3.31.1. Show error wp:show error
- In wp-config.php insert this before
/* That's all, stop editing! Happy blogging */ - https://codex.wordpress.org/Debugging_in_WordPress
if ( ! defined( 'WP_DEBUG' ) ) { if ( php_sapi_name() == 'cli' || ( isset( $_SERVER['PANTHEON_ENVIRONMENT'] ) && $_ENV['PANTHEON_ENVIRONMENT'] === 'live' ) ) { // Force to turn off debug for Live environment and WP-CLI define( 'WP_DEBUG', false ); } elseif ( 1 == 1 ) { // toggle debug define( 'WP_DEBUG', true ); define( 'WP_DEBUG_DISPLAY', true ); // set to false to not display error on HTML pages define( 'SAVEQUERIES', true ); // enable global $wpdb->queries to analyze queries @error_reporting( E_ALL ); @ini_set( 'log_errors', true ); @ini_set( 'log_errors_max_len', '0' ); @ini_set( 'display_errors', 1 ); // 'stderr' can be set // define( 'WP_DEBUG_LOG', true ); // /wp-content/debug.log useful to debug AJAX or wp-cron run // define( 'CONCATENATE_SCRIPTS', false ); // by default, for Admin area, WP loads one JavaScript. If you have one error in script, it will fail // set to false that JavaScript files are loaded separately for Admin area. } else { define( 'WP_DEBUG', false ); } }
Or anywhere
error_reporting(E_ALL); ini_set('display_errors', 1);
Write to file wp-content/debug.log after define( 'WP_DEBUG_LOG' , true );
error_log('Hello lili'); error_log(print_r($abc, 1));
3.31.2. Debug WP_Query
About SAVEQUERIES
if ( current_user_can( 'administrator' ) ) { global $wpdb; echo "<pre>"; print_r( $wpdb->queries ); echo "</pre>"; }
$query = new WP_Query($args); // debug: actual SQL raw query that is run echo "<pre>"; print_r($query->request); // to see what query vars are passed to wp_query // print_r($query->query_vars); echo "</pre>"; $args = array( 'post_type' => array('news'), // 'tax_query' => array(...), // ... 'lili' => 'lili' ); // inside class-wp-query.php function get_posts() if (isset($q['lili'])) { //var_dump($where); }
3.31.3. Which template? wp:global:template
3.31.4. All Image Sizes
add_action( 'wp_footer', 'debug_all_image_sizes' ); function debug_all_image_sizes() { print '<h1>All Image Sizes</h1>'; print '<pre>'; global $_wp_additional_image_sizes; print_r( $_wp_additional_image_sizes ); // [ 'image-size-name' => [ 'width' => 300, 'height' => 250, 'crop' => 1 ] ] // crop is empty means the image size is not croppable print '</pre>'; } function debug_template_file() { print '<h1>All Image Sizes</h1>'; print '<pre>'; global $template; print_r( $template ); print '</pre>'; }
3.31.5. Filters
if ( WP_DEBUG && array_intersect( array( 'administrator' ), wp_get_current_user()->roles ) ) { add_action( 'wp_footer', 'debug_genesis_attr_filters' ); } function debug_genesis_attr_filters() { global $wp_filter; // current_filter() might be a better way to do this $genesis_attr_filters = array(); $h1 = '<h1>Current Page Genesis Attribute Filters</h1>'; $out = ''; $ul = '<ul>'; foreach ( $wp_filter as $key => $val ) { if ( false !== strpos( $key, 'genesis_attr' ) ) { $genesis_attr_filters[ $key ][] = var_export( $val, true ); } } foreach ( $genesis_attr_filters as $name => $attr_vals ) { $out .= "<h2 id=$name>$name</h2><pre>" . implode( "\n\n", $attr_vals ) . '</pre>'; $ul .= "<li><a href='#$name'>$name</a></li>"; } print "$h1$ul</ul>$out"; }
global $wp_filter; print '<pre>'; // all filters // print_r($wp_filter); // a specific filter // print_r($wp_filter['posts_where']); // More advanced: find file name and line number of the callback function // array( // filter 1 // array( id, priority, function, accepted_args, file, line ) // ) print_r(list_hooks('posts_where')); print '</pre>'; // https://stackoverflow.com/a/26680808/2196360 function list_hooks( $hook = '' ) { global $wp_filter; if ( isset( $wp_filter[ $hook ]->callbacks ) ) { array_walk( $wp_filter[ $hook ]->callbacks, function ( $callbacks, $priority ) use ( &$hooks ) { foreach ( $callbacks as $id => $callback ) { $hooks[] = array_merge( [ 'id' => $id, 'priority' => $priority ], $callback ); } } ); } else { return []; } foreach ( $hooks as &$item ) { // skip if callback does not exist if ( ! is_callable( $item['function'] ) ) { continue; } // function name as string or static class method eg. 'Foo::Bar' if ( is_string( $item['function'] ) ) { $ref = strpos( $item['function'], '::' ) ? new ReflectionClass( strstr( $item['function'], '::', true ) ) : new ReflectionFunction( $item['function'] ); $item['file'] = $ref->getFileName(); $item['line'] = get_class( $ref ) == 'ReflectionFunction' ? $ref->getStartLine() : $ref->getMethod( substr( $item['function'], strpos( $item['function'], '::' ) + 2 ) )->getStartLine(); // array( object, method ), array( string object, method ), array( string object, string 'parent::method' ) } elseif ( is_array( $item['function'] ) ) { $ref = new ReflectionClass( $item['function'][0] ); // $item['function'][0] is a reference to existing object $item['function'] = array( is_object( $item['function'][0] ) ? get_class( $item['function'][0] ) : $item['function'][0], $item['function'][1] ); $item['file'] = $ref->getFileName(); $item['line'] = strpos( $item['function'][1], '::' ) ? $ref->getParentClass()->getMethod( substr( $item['function'][1], strpos( $item['function'][1], '::' ) + 2 ) )->getStartLine() : $ref->getMethod( $item['function'][1] )->getStartLine(); // closures } elseif ( is_callable( $item['function'] ) ) { $ref = new ReflectionFunction( $item['function'] ); $item['function'] = get_class( $item['function'] ); $item['file'] = $ref->getFileName(); $item['line'] = $ref->getStartLine(); } } return $hooks; }
3.32. Installation Language
Download additional language packs from here Upload thme to wp-content/languages Then change the language in each User Profile.
3.33. File Permissions
# set permission 644 on files find . -type f -exec chmod 644 {} + # set permissions 755 for directories find . -type d -exec chmod 755 {} + chmod 660 wp-config.php
Run this to check which files are not 644 and which directories are not 755 wp:check file permissions
find . -type d -not -perm 755 -o -type f -not -perm 644 find . -type d -not -perm 755 -not -path "./.git/*" -o -type f -not -perm 644 -not -path "./.git/*"
Multiple users upload as www-data linux:permission:users
3.34. SSL
// Force HTTPS logins and administration define('FORCE_SSL_ADMIN', true); // The above also means this below //define('FORCE_SSL_LOGIN', true); define('WP_HOME','https://'.$_SERVER[HTTP_HOST]); define('WP_SITEURL','https://'.$_SERVER[HTTP_HOST]);
You can now try to go to wp-admin If W3 Total Cache is installed, Performance > Page Cache > enable Cache SSL (https) requests If some security plugin is installed, make sure no SSL is enforced for any pages because Apache at the end will cover that.
Use ssl:test tools to identify problematic links Fix legacy content in DB https://css-tricks.com/moving-to-https-on-wordpress/
-- double quotes UPDATE wp_posts SET post_content = ( Replace (post_content, 'src="http://', 'src="//') ) WHERE Instr(post_content, 'jpeg') > 0 OR Instr(post_content, 'jpg') > 0 OR Instr(post_content, 'gif') > 0 OR Instr(post_content, 'png') > 0; -- single quote UPDATE wp_posts SET post_content = ( Replace (post_content, "src='http://", "src='//") ) WHERE Instr(post_content, 'jpeg') > 0 OR Instr(post_content, 'jpg') > 0 OR Instr(post_content, 'gif') > 0 OR Instr(post_content, 'png') > 0; -- Custom Fields UPDATE wp_postmeta SET meta_value=(REPLACE (meta_value, 'iframe src="http://','iframe src="//'));
Force redirect to https on Apache for all pages :: apache:https To setup nginx as host and apache as proxy nginx:proxy:docker
If WordPress is behind Nginx proxy server or load balancer, the above is not enough because is_ssl function will not work Inspired by plugin ssl-insecure-content-fixer wp-config.php
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && strtolower($_SERVER['HTTP_X_FORWARDED_PROTO']) === 'https') { $_SERVER['HTTPS'] = 'on'; }
3.35. REST API wp:rest
3.35.1. Methods (Also called operations, verbs, routes)
/tickets- is called endpoint
- GET /tickets
- Retrieves a list of tickets
- OPTIONS /tickets
- see any parameters and fields can be used for endpoint
tickets - GET /tickets/12
- Retrieves a specific ticket
- POST /tickets
- Creates a new ticket
- PUT /tickets/12
- Updates ticket #12
- PATCH /tickets/12
- Partially updates ticket #12
- DELETE /tickets/12
- Deletes ticket #12
3.35.2. Status Codes
3.35.2.1. 1xx Transfer protocol-level info
- 100
- An interim response. Indicates the client that the initial part of the request has been received and has not yet been rejected by the server. The client should continue by sending the remaminder of the request, or if the request has already been completed, ignore this response. The server MUST send a final response after the request has been completed
- 102
- Processing (WebDav). Indicates the server has received and is processing the request, but no response is available yet
3.35.2.2. 2xx Success
- 200
- OK. Should have a response body
- 201
- Created. Should return URI of the newly created resource
- 202
- Accepted. Used for actions that take a long while to process. Indicates the request has been accepted for processing but it's not completed. The request might or might not be eventually acted upon, or even maybe disallowed when processing occurs. Should return the request's current status and either a pointer to a status monitor (job queue location) or some estimate of when the user can expect the request to be fulfilled
- 203
- Non-authoritative information. The returned metainformation in the entity-header is not the definitive set as available from the origin server, but is gathered from a local or a third-party copy. The set presented MAY be a subset or superset of the original version
- 204
- No content
- Usually it's sent out in response to PUT, POST or DELETE request when REST API declines to send back any status message or representation in the response body
- In response to GET request to indicate that the requested resource exists but has no state representation to include in the body
- Must not include a message-body and thus is always terminated by the first empty line after the header fields
3.35.2.3. 3xx Redirection - The client must take some additional action in order to complete their request
- 300
- multiple choices. The request has more than one possible response. The user-agent or user should choose one of them
- 301
- Moved permanently. The URL of the requested resource has been changed permanently. The new URL is given by the Location header field in the resposne. This response is cachable unless indicated otherwise
- 302
- Found. The URL of the requested resource has been changed temporarily. The new URL is given by the Location field in the response. This response is only cacheable if indicated by a Cache-Control or Expires header field
- 303
- See other. The response can be found under a different URI and SHOULD be retrieved using a GET method on that resource
- 304
- Not modified. The client that the response has not been modified, so the client can continue to use the same cached version of the response
- 307
- Temporary redirect. The REST API is not going to process the client's request. Instead, the client should resubmit the request to the URI specified by the response message's Location header. However, future requests should still use the original URI. REST API can use this status code to assign a temporary URI to the client's requested resource. E.g. 307 response can be used to shift a client request over to another host.
- The temporary URI SHOULD be given by the Location field in the response. Unless the request method was HEAD, the entity of the response SHOULD contain a short hypertext note with a hyperlink to the new URI(s). If the 307 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued
3.35.2.4. 4xx Client error
- 400
- Bad request. General purpose when no other 4xx code is appropriate. The client SHOULD NOT repeat the request without modifications
- 401
- Unauthorized. MUST include a WWW-Authenticate header field containing a challenge applicable to the requested resource. The client MAY repeat the request which a suitable Authorization header field. If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials. If the 401 response contains the same challenge as the prior response, and the user agent has already attempted authentication at least once, then the user SHOULD be presented the entity that was given in the response, since that entity might include relevant diagnostic information
- 403
- Forbidden. The client request is formed correctly, but the REST API refuses to honor it e.g. the user does not have the necessary permissions for the resource. It's not a case of insufficient client credentials, that would be 401
- Authentication will not help, and the request SHOULD NOT be repeated
- 405
- Method not allowed. Must include the Allow header, which lists the HTTP methods that the resource supports e.g.
Allow: GET, POST - 406
- Not acceptable. A client request for data formatted as application/xml will receive 406 if the API is only willing to format data as application/json. API is not able to generate any of the client's preferred media types as indicated by the Accept request header. A user agent should temporarily stop receipt of more data and query the user for a decision on further actions
- 412
- Precondition failed. The client specified one or more preconditions in its request headers, effectively telling the REST API to carry out its request only if certain conditions were met
- 415
- Unsupported media type. API is not able to process tthe client's supplied media type, as indicated by the Content-Type request header
3.35.2.5. 5xx Server error
- 500
- Generic. e.g. when the server raises an exception
- 501
- Not implemented. The server either does not recognize the request method, or it cannot fulfill the request. Usually implies future availability
3.35.3. Global parameters
3.35.3.1. _jsonp
<script> function receiveData( data ) { // Do something with the data here. // For demonstration purposes, we'll simply log it. console.log( data ); } </script> <script src="https://demo.wp-api.org/wp-json/?_jsonp=receiveData"></script>
3.35.3.2. _method
This is for server that couldn't fire a DELETE HTTP request A POST to /wp-json/wp/v2/posts/42?_method=DELETE would be translated to a DELETE to the wp/v2/posts/42 route.
Similarly, the following POST request would become a DELETE:
POST /wp-json/wp/v2/posts/42 HTTP/1.1 Host: example.com X-HTTP-Method-Override: DELETE
3.35.3.3. _envelope
Some servers, clients, and proxies do not support accessing the full response data. The API supports passing an _envelope parameter, which sends all response data in the body, including headers and status code.
GET to wp/v2/users/me:
HTTP/1.1 302 Found
Location: http://example.com/wp-json/wp/v2/users/42
{
"id": 42,
...
}
GET to wp/v2/users/me?_envelope:
HTTP/1.1 200 OK
{
"status": 302,
"headers": {
"Location": "http://example.com/wp-json/wp/v2/users/42"
},
"body": {
"id": 42
}
}
3.35.3.4. _embed
- Refer to wp:rest:posts:embed
3.35.4. Pagination and Ordering
?page=- default 10, possible from 1 to 100
?offset=- asc desc
- date author relevance id include modified parent relevance slug include_slugs title
- By default, there's no rand! (random order)
- Refer to wp:rest:hook:rest_*_collection_params for adding rand
3.35.5. Response Header fields
X-WP-TOTAL- total number of records
- (no term)
X-WP-TotalPages
3.35.6. Endpoints
- To see all endoints
Method:OPTIONS http://mysite.io/wp-json/wp-v2/
3.35.6.1. Plugin rest-filter
Use https://github.com/wp-api/rest-filter to build more complex queries
Term slug of taxonomy news-category
wp-json/wp/v2/news?per_page=5&filter[news-category]=news-abc
3.35.6.2. Custom Post Types and Taxonomies
- Make sure both post type and taxonomy has
show_in_rest=truein wp:f:register_post_type~ and wp:f:register_taxonomyrest_base- change the base url of REST API route
rest_controller_class- Default is
WP_REST_Posts_Controller
- Core cannot query term slug
wp-json/wp/v2/news?per_page=5&news-category=1234- Multiple terms
wp-json/wp/v2/news?per_page=5&news-category=1234,4567
- Multiple terms in multiple taxonomies
wp-json/wp/v2/news?per_page=5&news-category=1234,4567&abc-category=789,123
3.35.6.3. posts
- https://developer.wordpress.org/rest-api/reference/posts/
- key
_links- has
self,collection,aboutandauthor, which are REST API links to the associated items of that post- wp:rest:posts:embed Some keys like
authorhasembeddableequals true, which means URL parameter_embed=truecan be used so that the post will include full post of author under_embedded
- wp:rest:posts:embed Some keys like
3.35.6.4. Custom endpoint
- Simple way
// http://example.com/wp-json/myplugin/v1/author/(?P<id>\d+) add_action( 'rest_api_init', function () { // version 1 register_rest_route( 'myplugin/v1', '/author/(?P<id>\d+)', array( 'methods' => 'GET', 'callback' => 'my_awesome_func', /*' optional permission_callback' => function () { return current_user_can( 'edit_others_posts' ); },*/ 'args' => array( 'id' => array( // 'default' => 'some default value', // 'sanitize_callback' => function() {}, // 'sanitize_callback' => 'absint', 'validate_callback' => function ( $param, $request, $key ) { return is_numeric( $param ); } ), ), ) ); } ); /** * Grab latest post title by an author! * * @param array $data Options for the function. * * @return string|null Post title for the latest, * or null if none. */ function my_awesome_func( WP_REST_Request $data ) { $posts = get_posts( array( 'author' => $data['id'], ) ); if ( empty( $posts ) ) { return new WP_Error( 'awesome_no_author', 'Invalid author', array( 'status' => 404 ) ); } $data = $posts[0]->post_title; // $data = array( 'some', 'response', 'data' ); // Direct return response // turn data into a WP_REST_Response object // return rest_ensure_response($data); // Or return a WP_REST_Response object // Create the response object $response = new WP_REST_Response( $data ); // Add a custom status code $response->set_status( 201 ); // Add a custom header //$response->header( 'Location', 'http://example.com/' ); return $response; }
- Using controller
- https://github.com/elevati/wp-api-multiple-posttype
/wp-json/wp/v2/multiple-post-type?&type[]=post&type[]=page- global search on
WP_REST_Controllerto see example e.g.WP_REST_Posts_Controller - Controller can do CRUD
add_action( 'rest_api_init', 'init_wp_rest_multiple_posttype_endpoint' ); function init_wp_rest_multiple_posttype_endpoint() { if ( ! class_exists( 'WP_REST_Multiple_PostType_Controller' ) ) { require_once dirname( __FILE__ ) . '/lib/endpoints/class-wp-rest-multiple-posttype-controller.php'; } $controller = new WP_REST_Multiple_PostType_Controller(); $controller->register_routes(); }
theme_path/lib/endpoints/class-wp-rest-multiple-posttype-controller.php
3.35.6.5. Batch
https://developer.wordpress.org/rest-api/requests/
- Batch run multiple REST API GET requets
- One HTTP GET request but return multiple responses as different key in JSON response
// example.com/wp-json/my-namespace/v1/batch?requests[0][method]=GET&requests[0][route]=/wp/v2/news&requests[0][params][per_page]=6&requests[0][params][orderby]=rand&requests[0][params][filter][center]=game-changer // /wp-json/wp/v2/news?per_page=6&filter[centre]=game-changer&orderby=rand // response // { 'GET /wp/v2/news': [ response for that REST API v2 request ]} // Register our mock batch endpoint. function prefix_register_batch_route() { register_rest_route( 'my-namespace/v1', '/batch', array( // Supported methods for this endpoint. WP_REST_Server::READABLE translates to GET. 'methods' => WP_REST_Server::READABLE, // Register the callback for the endpoint. 'callback' => 'prefix_do_batch_request', // Register args for the batch endpoint. 'args' => prefix_batch_request_parameters(), ) ); } add_action( 'rest_api_init', 'prefix_register_batch_route' ); /** * Our registered endpoint callback. Notice how we are passing in $request as an argument. * By default, the WP_REST_Server will pass in the matched request object to our callback. * * @param WP_REST_Request $request The current matched request object. */ function prefix_do_batch_request( $request ) { // Here we initialize the array that will hold our response data. $data = array(); $data = prefix_handle_batch_requests( $request['requests'] ); return $data; } /** * This handles the building of the response for the batch requests we make. * * @param array $requests An array of data to build WP_REST_Request objects from. * @return WP_REST_Response A collection of response data for batch endpoints. */ function prefix_handle_batch_requests( $requests ) { $data = array(); // Foreach request specified in the requests param run the endpoint. foreach ( $requests as $request_params ) { $response = prefix_handle_request( $request_params ); $key = $request_params['method'] . ' ' . $request_params['route']; $data[ $key ] = prefix_prepare_for_collection( $response ); } return rest_ensure_response( $data ); } /** * This handles the building of the response for the batch requests we make. * * @param array $request_params Data to build a WP_REST_Request object from. * @return WP_REST_Response Response data for the request. */ function prefix_handle_request( $request_params ) { $request = new WP_REST_Request( $request_params['method'], $request_params['route'] ); // Add specified request parameters into the request. if ( isset( $request_params['params'] ) ) { foreach ( $request_params['params'] as $param_name => $param_value ) { $request->set_param( $param_name, $param_value ); } } $response = rest_do_request( $request ); return $response; } /** * Prepare a response for inserting into a collection of responses. * * This is lifted from WP_REST_Controller class in the WP REST API v2 plugin. * * @param WP_REST_Response $response Response object. * @return array Response data, ready for insertion into collection data. */ function prefix_prepare_for_collection( $response ) { if ( ! ( $response instanceof WP_REST_Response ) ) { return $response; } $data = (array) $response->get_data(); $server = rest_get_server(); if ( method_exists( $server, 'get_compact_response_links' ) ) { $links = call_user_func( array( $server, 'get_compact_response_links' ), $response ); } else { $links = call_user_func( array( $server, 'get_response_links' ), $response ); } if ( ! empty( $links ) ) { $data['_links'] = $links; } return $data; } /** * Returns the JSON schema data for our registered parameters. * * @return array $params A PHP representation of JSON Schema data. */ function prefix_batch_request_parameters() { $params = array(); $params['requests'] = array( 'description' => esc_html__( 'An array of request objects arguments that can be built into WP_REST_Request instances.', 'my-text-domain' ), 'type' => 'array', 'required' => true, 'validate_callback' => 'prefix_validate_requests', 'items' => array( array( 'type' => 'object', 'properties' => array( 'method' => array( 'description' => esc_html__( 'HTTP Method of the desired request.', 'my-text-domain' ), 'type' => 'string', 'required' => true, 'enum' => array( 'GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', ), ), 'route' => array( 'description' => esc_html__( 'Desired route for the request.', 'my-text-domain' ), 'required' => true, 'type' => 'string', 'format' => 'uri', ), 'params' => array( 'description' => esc_html__( 'Key value pairs of desired request parameters.', 'my-text-domain' ), 'type' => 'object', ), ), ), ), ); return $params; } function prefix_validate_requests( $requests, $request, $param_key ) { // If requests isn't an array of requests then we don't process the batch. if ( ! is_array( $requests ) ) { return new WP_Error( 'rest_invald_param', esc_html__( 'The requests parameter must be an array of requests.' ), array( 'status' => 400 ) ); } foreach ( $requests as $request ) { // If the method or route is not set then we do not run the requests. if ( ! isset( $request['method'] ) || ! isset( $request['route'] ) ) { return new WP_Error( 'rest_invald_param', esc_html__( 'You must specify the method and route for each request.' ), array( 'status' => 400 ) ); } if ( isset( $request['params'] ) && ! is_array( $request['params'] ) ) { return new WP_Error( 'rest_invald_param', esc_html__( 'You must specify the params for each request as an array of named key value pairs.' ), array( 'status' => 400 ) ); } } // This is a black listing approach to data validation. return true; }
3.35.7. Authentication
Require auth for all requests
add_filter( 'rest_authentication_errors', function( $result ) { if ( ! empty( $result ) ) { return $result; } if ( ! is_user_logged_in() ) { return new WP_Error( 'rest_not_logged_in', 'You are not currently logged in.', array( 'status' => 401 ) ); } return $result; });
3.35.8. Current request is REST request?
if ( defined('REST_REQUEST') && REST_REQUEST) {}
3.35.9. New internal request
$request = new WP_REST_Request( 'GET', '/wp/v2/posts' ); // Set one or more request query parameters $request->set_param( 'per_page', 20 ); $response = rest_do_request( $request );
3.35.10. New external request and read response
$request = wp_remote_get( 'https://a.ca/wp-json/wp/v2/news?per_page=5' ); if ( is_wp_error( $request ) ) { return false; // Bail early } $body = wp_remote_retrieve_body( $request ); $data = json_decode( $body ); if ( ! empty( $data ) ) { echo $data[0]->title->rendered; }
3.35.11. Cache
I think REST response is object cache. When using with Pantheon Advanced Page Cache and WP Redis plugins, REST response is cached and it's refreshed when necessary after post CRUD.
3.35.12. Hooks
3.35.12.1. rest_api_init
register_rest_field()Add an extra field in response
add_action( 'rest_api_init', 'll_rest_api_add_fields' ); function ll_rest_api_add_fields() { //Add featured image register_rest_field( array('product'), // Where to add the field (Here, blog posts. Could be an array) 'featured_image_src', // Name of new field (You can call this anything) array( 'get_callback' => function( $object, $field_name, $request ) { $feat_img_array = wp_get_attachment_image_src( $object['featured_media'], // Image attachment ID 'thumbnail', // Size. Ex. "thumbnail", "large", "full", etc.. true // Whether the image should be treated as an icon. ); return $feat_img_array[0]; }, 'update_callback' => null, 'schema' => null, ) ); // add some custom fields register_rest_field( 'product', 'post_meta_fields', array( 'get_callback' => function( $post ) { $fields = [ 'image' => get_field( 'product_image' )['url'], 'product_categories' => (array) wp_get_post_terms( $post['id'], 'product_category' ), 'company' => (string) get_field( 'company_name', $post['id'] ), 'booth' => (string) get_field( 'booth_number', $post['id'] ), ]; return $fields; }, ) ); }
3.35.12.2. rest_after_insert_{post_type}
- Default no action is added to the hook
- Action runs when task post is modified via REST API, Gutenberg or Post editor
add_action( 'rest_after_insert_task', 'taskbook_change_status', 10, 2 ); function taskbook_change_status( $post, $request ) { $outcome = get_post_meta( $post->ID, 'taskbook_outcome', true ); if ( 0 === strlen($outcome) ) { update_post_meta( $post->ID, 'task_status', 'In progress' ); } else { update_post_meta( $post->ID, 'task_status', 'Completed' ); } }
3.35.12.3. rest_{$this->post_type}_collection_params wp:rest:hook:rest_*_collection_params
- Register custom collection parameters
$_GET['yourparam'] - Add options to existing parameter e.g. add
randtoorderby - For
post, userest_post_collection_params
// add rand to orderby for post type news add_filter( 'rest_news_collection_params', 'my_prefix_add_rest_orderby_params', 10, 1 ); function my_prefix_add_rest_orderby_params( $params ) { $params['orderby']['enum'][] = 'rand'; return $params; }
3.35.13. Basic Auth
- https://github.com/WP-API/Basic-Auth
Enable WP API to authenticate using
Authorization: Basic base64HasedOfUsername:Password. To submit request from client:curl --user admin:password https://example.com/wp-json/
$args = array( 'headers' => array( 'Authorization' => 'Basic ' . base64_encode( $username . ':' . $password ), ), );
3.35.14. Nonce
$.ajax({
url: WPsettings.root + 'wp/v2/posts/' + WPsettings.current_ID,
method: 'POST',
beforeSend: function(xhr) {
xhr.setRequestHeader( 'X-WP-Nonce', WPsettings.nonce);
},
data: {
'title': newTitle
}
})
3.35.15. JWT - JSON Web Token
- Plugin
- https://en-ca.wordpress.org/plugins/jwt-authentication-for-wp-rest-api/
Add auth endpoints
- POST /wp-json/jwt-auth/v1/token with raw JSON in body
{ "username": "", "password": "" }
- Token is retrieved. Then in subsequent requests include this header
Authorization: Bearer $token
- POST /wp-json/jwt-auth/v1/token with raw JSON in body
Need to add to wp-config.php
define('JWT_AUTH_SECRET_KEY', 'your-top-secret-key'); define('JWT_AUTH_CORS_ENABLE', true);
Enable HTTP Authroization header, e.g. Apache
RewriteEngine on RewriteCond %{HTTP:Authorization} ^(.*) RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1]
3.36. SOAP vs RESTful API
- SOAP
- Simple Object Access Protocol
- only allows XML
- Used to expose components of application logic as service rather than data
- perform operations through a standardized set of messaging patterns
Example 1: getMessage request to SOAP server
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"> <soap:Header> <wsse:Security soap:mustUnderstand="1"> <wsse:UsernameToken> <wsse:Username>%%wsse:Username%%</wsse:Username> <wsse:Nonce>%%wsse:Nonce%%</wsse:Nonce> <wsu:Created>%%wsu:Created%%</wsu:Created> <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">%%wsse:Password%%</wsse:Password> </wsse:UsernameToken> </wsse:Security> </soap:Header> <soap:Body> <getMessage xmlns="http://soap.server.ca"> <channel>%%linkid%%</channel> </getMessage> </soap:Body> </soap:Envelope>
- Link ID or Channel ID: each channel may have multiple different users
- Nonce is completely random for each request
- Password is hashed with time and the real password. Related to
wsu:Created
- REST
- Representational State Transfer
- Used to access data (named resources)
3.37. Life Cycle
3.37.1. index.php
- index.php
- WP_USE_THEMES = true
wp-blog-header.php
- wp-load.php wp:wp-load.php
- wp:global:wpdb
- wp-config.php
wp-settings.php
- wp-includes/load.php
- wp-includes/default-constants.php
- wp:api:plugin
- wp-includes/compat.php (functions)
- wp-includes/class-wp-list-util.php
- wp-includes/formatting.php
- wp:api:main
- wp-includes/option.php
- wp:api:option
- wp-includes/class-wp-matchesmapregex.php
- wp-includes/class-wp.php (wp environment setup class)
- wp-includes/class-wp-error.php (Error API)
- wp-includes/pomo/mo.php (class for working with MO files)
- translations.php
- streams.php
- require_wp_db()
- wp-db.php
- wp-includes/default-filters.php
- L10n library
- wp-includes/l10n.php
- wp-includes/class-wp-locale.php
- wp-includes/class-wp-locale-switcher.php
- wp:api:role-capabilities
- wp:api:query
- template loading functions
- wp-includes/general-template.php
- wp:api:post
- wp:api:rewrite
- wp:api:feed
- wp:api:dependency
- Core Taxonomy API wp:api:taxonomy
- wp:api:shortcode
- wp:api:oembed
- wp:api:media
- wp:api:http request
- wp:api:widgets
- wp-includes/nav-menu.php
- wp:api:toolbar
- Load mu-plugins
- wp:action:mu_plugin_loaded
- wp:action:muplugins_loaded
- wp:f:register_taxonomy wp:f:register_post_type
- Load active plugins
- wp:action:plugin_loaded
- wp:pluggable
- wp:action:plugins_loaded
- Initialize objects
$GLOBALS['wp_the_query'] = new WP_Query();$GLOBALS['wp_query'] = $GLOBALS['wp_the_query']$GLOBALS['wp_rewrite'] = new WP_Rewrite();
- functions.php
- wp:action:after_setup_theme
do_action( 'init' )do_action( 'wp_loaded' )
wp( $query_vars = '' );$wp->main( $query_vars )class-wp.php
wp_get_current_user()- wp:wp:m:parse_request
- wp:filter:do_parse_request
- short circuit
- wp:filter:query_vars
apply_filters( 'query_vars', $this->public_query_vars )- (no term)
Parse and assign values for each public_query_vars as
$wpvar.- Set
$wp->query_vars[ $wpvar ]to value in highest order:
$wp->extra_query_vars[ $wpvar ]$_POST[ $wpvar ]$_GET[ $wpvar ]$perma_query_varsfrom wp_rewrite match- If
$wpvaris a publicly_queryable wp:f:register_post_type:query_var, add query vars
- If
- post type slug
$wp->query_vars[ $wpvar ]
- Set
- (no term)
- Sanitize taxonomy and post type query vars
- (no term)
- Parse private_query_vars from $extra_query_vars
- wp:filter:request
apply_filters( 'request', $this->query_vars )- wp:action:parse_request
do_action_ref_array( 'parse_request', array( &$this ) )
$wp->send_headers()- wp:action:send_headers
do_action_ref_array( 'send_headers', array( &$this ) );
$wp->query_posts()$wp->build_query_string();- wp:filter:query_string
apply_filters( 'query_string', $this->query_string )
- wp:wp_query:m:query
- wp:action:wp">
- wp-includes/template-loader.php
- wp:action:template_redirect
- wp:action:do_robots
- wp:filter:template_include
3.37.2. wp-admin/index.php
- wp-admin/index.php
- wp-admin/admin.php
- wp:wp-load.php
- Core Administration API wp:api:core admin
- wp-admin/includes/dashboard.php
- wp-admin/admin-header.php
- wp-admin/admin-footer.php
- wp-admin/admin.php
3.37.3. wp-admin/admin-ajax.php
- Request wp-admin/admin-ajax.php
- /wp-load.php
- /wp-config.php
- /wp-settings.php (core files, active plugins and themes and the REST API)
- /wp-admin/admin.php
- /wp-admin/includes/ajax-actions.php
3.37.4. Request REST API
- Request REST API
index.php
- wp-blog-header.php (Environment and Template)
- wp-load.php
- wp-config.php
- wp-settings.php (which loads most core files, all active plugins and themes, and the REST API) (doesn't load admin_init,
/wp-admin/*.*)
3.38. PHP 7
Notice: The called constructor method for WP_Widget is deprecated since version 4.3.0! Use __construct() instead. in /path-to/wp-includes/functions.php on line 3457- Disable Notice logging
- wp:filter:deprecated_constructor_trigger_error
- Or fix the code
- wp:f:register_widget
Your PHP installation appears to be missing the MySQL extension which is required by WordPress
3.39. Tools
- wpscans.com
3.40. UC: Simple HTML page with form submit
Create a url path /abc which points to index.html (can't have php files)
3.40.1. /abc/index.html
<script src='https://www.google.com/recaptcha/api.js'></script> <div id="form-frame" class="form"> <div class="form-input"> <div id="err-message"></div> </div> <div class="form-input"> <label for="li-fname">Full Name</label> <input type="text" name="li-fname" id="li-fname"> </div> <div class="form-input news-input"> <label for="li-sub-promo">How did you hear about this promotion?</label> <select name="li-sub-promo" id="li-sub-promo"> <option value="Website">Website</option> <option value="Radio">Radio</option> <option value="TV">TV</option> <option value="Referral">Referral</option> <option value="Brochure">Brochure</option> <option value="Email">Email</option> <option value="Newsletter">Newsletter</option> <option value="Online Ads">Online Ads</option> <option value="Social Media">Social Media</option> </select> </div> <div class="form-input news-input"> <label for="small-business-lighting">Programs of Interest <br>(Check all that apply)</label> <div class="li-newsbox"> <input type="checkbox" name="li-newsbox" id="small-business-lighting">Small Business Lighting<br> <input type="checkbox" name="li-newsbox" id="business-registration-incentives">Business Refrigeration Incentives </div> </div> <div class="form-input"> <label> </label> <div class="g-recaptcha" data-sitekey="[[your_google_recaptcha_public_key]]"></div> <div id="captcha-verify"></div> </div> <div class="form-input submit-input" id="form-submit"> <label> </label> <a href="javascript:;" onclick="javascript:submit_form();">Submit</a> </div> </div> <script type="text/javascript" src="/abc/lb/js/app.js"></script> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
3.40.2. /abc/lb/js/app.js
function submit_form(){ var send_msg = document.getElementById('form-submit'); var pre_send = ('<label> </label><a href="javascript:;" onclick="javascript:submit_form();">Submit</a>'); var post_send = ('<label> </label><a id="send-blank">Sending...</a>'); var validate_form = pre_validate_cf(); var message = document.getElementById('err-message'); var form_data = ''; var form_return = ''; send_msg.innerHTML = post_send; if (validate_form!=0) { send_msg.innerHTML = pre_send; message.innerHTML = 'Please correct the following errors: <ul>'+validate_form+'</ul>'; message.style.display = 'block'; } else { message.style.display = 'none'; /* -- SEND TO SERVER -- */ var name = encodeURIComponent(document.getElementById('li-fname').value); var email = encodeURIComponent(document.getElementById('li-email').value); var phone = encodeURIComponent(document.getElementById('li-phone').value); var company = encodeURIComponent(document.getElementById('li-company').value); var address = encodeURIComponent(document.getElementById('li-address').value); var promo = encodeURIComponent(document.getElementById('li-sub-promo').value); var bus_lighting = is_checked('small-business-lighting'); var bus_reg_incentives = is_checked('business-registration-incentives'); form_data = 'name='+name+'&email='+email+'&phone='+phone+'&company='+company+'&address='+address+'&promo_hear='+promo+'&bus_lighting='+bus_lighting+'&bus_reg_incentives='+bus_reg_incentives; cf_form_req(form_data); } } function cf_form_req(params){ var send_msg = document.getElementById('form-submit'); var pre_send = ('<label> </label><a href="javascript:;" onclick="javascript:submit_form();">Submit</a>'); var post_send = ('<label> </label><a id="send-blank">Sending...</a>'); var message = document.getElementById('err-message'); var recaptcha = ''; recaptcha = grecaptcha.getResponse(); var params_full = params +'&recaptcha='+ recaptcha; send_msg.innerHTML = post_send; var url = "/wp-content/themes/li-cdm-lp/lb/api-localbusiness/cf-form-post.php"; $.ajax({ url : url, type: "POST", data : params_full, success: function(data) { console.log(data); if(data=='true'){ document.getElementById('form-validated').style.display = 'block'; ga('send', 'event', { eventCategory: 'Form Submit Success', eventAction: 'click', eventLabel: event.target.href }); console.log('submit'); }else { message.style.display = 'block'; send_msg.innerHTML = pre_send; message.innerHTML = 'Please correct the following errors: <ul>'+data+'</ul>'; } } }); } function google_captcha_verify(){ var v = grecaptcha.getResponse(); if(v.length == 0) { return false; } else { return true; } } function pre_validate_cf(){ var name = document.getElementById('li-fname'); var email = document.getElementById('li-email'); var phone = document.getElementById('li-phone'); var company = document.getElementById('li-company'); var address = document.getElementById('li-address'); var bus_lighting = is_checked('small-business-lighting'); var bus_reg_incentives = is_checked('business-registration-incentives'); var google_captcha = google_captcha_verify(); //var captcha = document.getElementById('captcha-verify'); var clean_css = 'border:1px #2ac158 solid'; var err_css = 'border:1px #FF0000 solid'; var check = ''; var log_data = ''; if(name.value == null || name.value==''){ check = '<li>Please enter your name.</li>'; log_data = '[ Empty First Name ],'; name.style.cssText = err_css; }else { log_data = '[ First Name: '+name.value+' ],'; name.style.cssText = clean_css; } if(email.value == null || email.value==''){ check += '<li>Please enter your email.</li>'; log_data += '[ Empty Email ],'; email.style.cssText = err_css; }else if(validate_email(email.value)=='false'){ check += '<li>Please enter a valid email.</li>'; log_data += '[ Invalid Email: '+email.value+' ],'; email.style.cssText = err_css; }else { log_data += '[ Email: '+email.value+' ],'; email.style.cssText = clean_css; } if(phone.value == null || phone.value==''){ check += '<li>Please enter your phone number.</li>'; phone.style.cssText = err_css; }else if(validate_phone(phone)!=true){ check += '<li>Please enter a valid phone number. Format: 000-000-0000</li>'; phone.style.cssText = err_css; }else { phone.style.cssText = clean_css; } if(company.value == null || company.value==''){ check += '<li>Please enter your company name.</li>'; company.style.cssText = err_css; }else { company.style.cssText = clean_css; } if(address.value == null || address.value == ''){ check += '<li>Please enter your company address</li>'; address.style.cssText = err_css; }else { address.style.cssText = clean_css; } if(google_captcha!== true ){ check += '<li>Please verify you are not a robot.</li>'; } if(bus_lighting!== 'Checked'){ if(bus_reg_incentives!=='Checked'){ check += '<li>Please select a program of interest.</li>'; }else { } } /* IF VALID -- Clear Check, Set to 0 */ if(check==''){check=0;} return check; } function is_checked(id){ if(document.getElementById(id).checked) { return( "Checked" ); }else { return( "Not Checked" ); } } function close_form(){ document.getElementById('form-validated').style.display = 'none'; }
3.40.3. /wp-content/themes/li-cdm-lp/lb/api-localbusiness/cf-form-post.php
<?php /* --- CLASS INCLUSION --- */ include('form-clean.php'); $FORMCLEAN = new FORM_CLEAN(); session_start(); $recipients_sbl = array('iamadmin@abc.com'); /* --- PARAMS (Pre-Clean) --- */ $C_NAME = $FORMCLEAN->clean($_POST['name']); $C_EMAIL = $FORMCLEAN->clean($_POST['email']); $C_PHONE = $FORMCLEAN->clean($_POST['phone']); $C_COMPANY = $FORMCLEAN->clean($_POST['company']); $C_ADDRESS = $FORMCLEAN->clean($_POST['address']); $C_PROMO = $FORMCLEAN->clean($_POST['promo_hear']); $C_BUS_LIGHTING = $FORMCLEAN->clean($_POST['bus_lighting']); $C_BUS_INCENTIVES = $FORMCLEAN->clean($_POST['bus_reg_incentives']); /* --- VALIDATE --- */ $C_NAME_VALIDATE = $FORMCLEAN->validate($C_NAME,'Name'); $C_EMAIL_VALIDATE = $FORMCLEAN->validate($C_EMAIL,'Email'); $C_PHONE_VALIDATE = $FORMCLEAN->validate($C_PHONE,'Phone'); $C_COMPANY_VALIDATE = $FORMCLEAN->validate($C_COMPANY,'Company'); $C_ADDRESS_VALIDATE = $FORMCLEAN->validate($C_ADDRESS,'Address'); // validate reCaptcha $C_CAPTCHA_VALIDATE = ''; $C_CAPTCHA = $_POST['recaptcha']; $C_CAPTCHA_google_request = array( 'url' => 'https://www.google.com/recaptcha/api/siteverify', 'options' => array( 'secret' => 'your_recaptcha_secret_key', 'response' => $C_CAPTCHA, )); $ch = curl_init(); $options = array( CURLOPT_URL => $C_CAPTCHA_google_request['url'], CURLOPT_POSTFIELDS => http_build_query($C_CAPTCHA_google_request['options']), CURLOPT_FOLLOWLOCATION => 1, CURLOPT_HEADER => 0, CURLOPT_RETURNTRANSFER => 1 ); curl_setopt_array($ch, $options); $C_CAPTCHA_google_result = json_decode(curl_exec( $ch ), true); curl_close($ch); if (!(is_array($C_CAPTCHA_google_result) && $C_CAPTCHA_google_result['success'] === true)) { $C_CAPTCHA_VALIDATE = ('<li>reCaptcha Match Fails </li>'); } // validate reCaptcha. $EMAIL_TO = ''; $SOURCE_CODE = 'Li-Local-Business'; $VALIDATE = str_replace('0','',$C_NAME_VALIDATE.$C_EMAIL_VALIDATE.$C_PHONE_VALIDATE.$C_COMPANY_VALIDATE.$C_ADDRESS_VALIDATE.$C_CAPTCHA_VALIDATE); if($VALIDATE==''){ if($C_BUS_LIGHTING=='Checked'){ /* --- SEND EMAIL (TO ADMINISTRATOR) --- */ $EMAIL_TO = implode(',', $recipients_sbl); //$EMAIL_TO .= 'joseph.alonzi@cleversamurai.com'; $SUBJECT = '[ Li ] - Small Business Lighting Signup'; $EMAIL_FROM = 'donotreply@abc.com'; $MESSAGE = (' <html> <head> <title>[ Li ] - Small Business Lighting Signup</title> </head> <body> <h1>[ Li ] - Small Business Lighting Signup</h1> <b>Name:</b> '.$C_NAME.'<br /> <b>Email:</b> '.$C_EMAIL.'<br /> <b>Phone:</b> '.$C_PHONE.'<br /><br /> <b>How Did You Hear About Us?</b> '.$C_PROMO.'<br /><br /> <b>Business Name:</b> '.$C_COMPANY.'<br /> <b>Business Address:</b> '.$C_ADDRESS.'<br /><br /> The following email: '.$C_EMAIL.' has shown interest in Small Business Lighting. <br /> --- <br /> Source Code: '.$SOURCE_CODE.$_debug.' </body> </html> '); $HEADERS = 'MIME-Version: 1.0' . "\r\n"; $HEADERS .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n"; $HEADERS .= 'From: Li <'.$EMAIL_FROM.'>' . "\r\n"; mail($EMAIL_TO, $SUBJECT, $MESSAGE, $HEADERS); } unset($_SESSION['C_CAPTCHA']); session_destroy(); /* --- IF ALL VALIDATES, AND THE EMAIL HAS BEEN SENT --- */ echo 'true'; } else { echo $VALIDATE; }
3.40.4. /wp-content/themes/li-cdm-lp/lb/api-localbusiness/form-clean.php
<?php class FORM_CLEAN{ public function clean($input){ /* -- Simple Clean -- */ $input = preg_replace('#<script(.*?)>(.*?)</script>#is', '', $input); $input = str_replace('<?php','',$input); $input = str_replace('?>','',$input); $input = str_replace("1=1;--",'',$input); $input = strip_tags($input); return $input; } public function validate($input,$name){ $error=0; if($input==''){ $error++; } if($error!=0){ return '<li> '.$name.' field is empty or invalid.'; }else { return $error; } } public function check_convert($str){ if($str=='true'){ $str='YES'; }else { $str='NO'; } return $str; } }
3.41. TS: Download failed. Destination directory for file streaming does not exist or is not writable
When updating plugins from wp-admin. wp requires the temp folder to be chmod:777
# create a temp folder with 777 permission outside of the webroot cd .. mkdir temp chmod 777 temp chown www-data:www-data
Add this line to wp-config.php
define('WP_TEMP_DIR', ABSPATH . '/../temp/');
3.42. TS: Hacked wp:ts:hacked
https://codex.wordpress.org/FAQ_My_site_was_hacked https://codex.wordpress.org/Hardening_WordPress Change wp users' passwords Compare wp-config.php with wp-config-sample.php wp:file permissions wp:wp-config:DISALLOW_FILE_EDIT
3.42.1. Search backdoor scripts
Use wp:plugin:exploit-scanner to check core and plugin files https://sucuri.net/guides/how-to-clean-hacked-wordpress Used PHP functions base64 str_rot13 gzuncompress eval exec system assert stripslashes preg_replace (with e) move_uploaded_file
3.42.2. Don't execute PHP in wp-content/uploads folder
In wp-content/uploads folder upload this .htaccess
<Files *.php> deny from all </Files>
Find *.php in uploads folder
find wp-content/uploads/ iname "*.php"
3.43. TS: wp_deregister_script was called incorrectly
Scripts and styles should not be registered or enqueued until the wp_enqueue_scripts, admin_enqueue_scripts, or login_enqueue_scripts hooks. Use something like this
if ( ! is_admin() ) { add_action( 'wp_enqueue_scripts', function(){ wp_deregister_script( 'jquery' ); wp_register_script( 'jquery', 'http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js', array(), null, false ); wp_enqueue_script( 'jquery'); }); }
3.44. TS: Image editor not working, Edit Media, PHP open tag
Ensure all php files in plugins, mu-plugins, theme's functions.php and its included PHP files:
- No space before
<?phpif the file starts with it - Files do not end with
?>
3.45. TS: Check list
- check PHP and MySQL version
- check WP Core version
- check proprietary plugins and see which plugins are deprecated. Check PHP and WP version required by plugins.
- check if there's custom/modified plugins
- check errors wp:show error
- check sitemap
- check website status
- wp-admin Settings > Reading > Search Engine Visibility > uncheck Discourage search engines from indexing this site.
4. ColdFusion
4.1. cfsetting
Override the server setting for a particular page
- requesttimeout
- can be overriden in
cfquery(database has to support) orcfhttpusing thetimeoutattribute in seconds
<cfsetting enablecfoutputonly="yes|no" requesttimeout="300" showdebugoutput="yes|no">
4.2. Text
FindNoCase("the", "The sentence") returns integer position
0 if not found
4.3. Mail
<cfmail to="1@a.com,2@b.com" from="" subject="" type="text/html">
<cfmailparam name="X-SMTPAPI" value='{"category": ["abc-xyz"]}' />
<p>...</p>
</cfmail>
4.4. Function
<cffunction name="getSiteDetail" access="public" returntype="query">
<cfargument name="intID" type="numeric" default="0">
<cfset var getSite = "">
<cfquery name="getSite" datasource="..." dbtype="ODBC">
SELECT *
FROM Site
WHERE ID <> 7
<cfif argument.intID neq 0>
AND ID = <cfqueryparam value="#Arguments.intID#" CFSQLType="CF_SQL_NUMERIC">
</cfif>
ORDER BY Site.sname
</cfquery>
<cfreturn getSite>
</cffunction>
<cfinvoke component="functions.banner"
method="getSiteDetail"
returnvariable="getSite">
<cfinvokeargument name="intID" value="#Form.ssite2#">
</cfinvoke>
4.5. Query
- Attributes
- timeout
- in seconds
<!--- Access ---> <cfquery name="queryname" datasource="abc" dbtype="ODBC"> SELECT * FROM tblOne </cfquery>
4.5.1. Get Identity After Insert
Access
<cftransaction>
<cfquery name="insertQ" datasource="abc">
INSERT INTO aTable (col1, col2)
VALUES (1,2)
</cfquery>
<cfquery name="getID" datasource="abc">
SELECT @@identity AS RecordID
</cfquery>
</cftransaction>
<cfoutput>#getID.recordID#</cfoutput>
TSQL
<cfquery> SET NOCOUNT ON; INSERT INTO aTable (col1,col2) VALUES (1,2) SELECT SCOPE_IDENTITY() AS RecordID; </cfquery>
4.5.2. Insert Null
<cfquery>
UPDATE tTable SET
<cfif isDefined('form.formFieldA')>
intField = <cfqueryparam value="#form.formFieldA#" cfsqltype="cf_sql_numeric">
<cfelse>
intField = <cfqueryparam value="" cfsqltype="cf_sql_numeric" null="1">
</cfif>
</cfquery>
4.6. Fast form to Database
Checkbox
<input type="checkbox" name="dLocked" value="1"> <cfparam name="form.dLocked" default="0" overwrite="false"> INSERT INTO aTable (dLocked) VALUES ( <cfqueryparam value="#form.dLocked#" CFSQLType="CF_SQL_BIT"> )
<!--- Insert --->
<form action="site_new.cfm" method="post">
<input name="Sname" type="text" size="30" maxlength="255">
</form>
// site_new.cfm
<cfinsert datasource="aDB" tablename="TableName">
<!--- Update. cfupdate doesn't work... use cfquery --->
<cfif isDefined("form.phone")>
<cfupdate datasource="cfdocexamples" tablename="EMPLOYEES">
</cfif>
<cfquery name="empTable" datasource="cfdocexamples">
SELECT * FROM EMPLOYEES
</cfquery>
<!--- This code shows the contents of the employee table and allows you to choose a row for updating. --->
<table border="1">
<cfoutput query="empTable">
<tr>
<td>#firstName#</td>
<td>#lastName#</td>
<td>#phone#</td>
<td><a href="cfupdate.cfm?id=#emp_id#">Edit</a></td>
</tr>
</cfoutput>
</table>
<cfif isDefined("url.id")>
<cfquery name="phoneQuery" datasource="cfdocexamples">
SELECT * FROM employees WHERE emp_id=#url.id#
</cfquery>
<!--- This code displays the row to edit for update. --->
<cfoutput query="phoneQuery">
<form action="cfupdate.cfm" method="post">
#phoneQuery.firstName# #phoneQuery.lastName#
<input name="phone" type="text" value="#phone#" size="12">
<input type="submit" value="Update">
<input name="emp_id" type="hidden" value="#emp_id#">
<!--- The emp_id is passed as a hidden field to be used as a primary
key in the CFUPDATE. --->
</form>
</cfoutput>
</cfif>
4.7. Loop
<cfloop query="aQueryname" startRow="1" endRow="10" group="Customer_ID"> #aQueryName.CurrentRow# </cfloop>
startRow, endRow and group are optional.
Array
<cfset local.objPath = GetPageContext().getRequest().getServletPath() /> <cfset local.apiNS = ArrayNew(2) /> <cfset local.apiNS[1] = ["v1", "json"] /> <cfset local.apiNS[2] = ["v2", "json2"] /> <cfset local.apiVer = 0 /> <cfset Request.apiCFC = 'json' /> <cfloop array="#local.apiNS#" item="i"> <cfif FindNoCase('/' & i[1] & '/', local.objPath, 1)> <cfset local.apiVer = i[1] /> <cfset Request.apiCFC = i[2] /> <cfbreak> </cfif> </cfloop>
4.8. Remote Address
<cfif cgi.REMOTE_ADDR eq "127.0.0.1">
4.9. cftry
<cftry>
<!--- Do something --->
<cfcatch type="any">
<cfdump var="#cfcatch#">
</cfcatch>
</cftry>
4.10. cfdump
Save cfdump in a string
<cfsavecontent variable="theDump"> <cfdump var="#form.uploadSizesAll#" expand="yes" format="text"> </cfsavecontent> <cfset sError = theDump>
4.11. Javascript
<cfoutput>
<script>
var #toscript(form.afield, "jsVarName")#;
</script>
</cfoutput>
5. Python
5.1. Installation
- Version
python -V- (no term)
- Windows
Choose the latest release on https://www.python.org/downloads/windows/
- And choose
Windows x86-64 executable installer - https://www.python.org/downloads/release/python-382/
- (no term)
- Run as Admin
- Add to PATH and choose Customize Installation to install it on
C:\Python\PythonVersionName
- And choose
5.2. Basics
print,format-
x = 42 print(f'Hello, {x}') # eq. to print('Hello, {}'.format(x)) x = 42 y = 73 if x < y: print('x < y: x is {} and y is {}'.format(x, y))
- (no term)
Condition
if x > y: print('x > y') elif x < y: print('x < y') elif x == y: print('x == y') else: print('this is unexpected..')
- (no term)
Loop
words = ['1', '2', '3'] n = 0; while(n < 5): print(words[n]) n += 1 a, b = 0, 1 while b < 1000: print(b, end = ' ', flush = True) a, b = b, a + b # a = b # b = a + b print() # line ending for i in words: print(i) # from 2 to n-1 for x in range(2, n): # do something
- (no term)
Function
def abc(n = 1): print(n) # return n x = abc(42) print(x) # every function return something, if not defined, it returns None
- (no term)
- Types
print(type(x))String
x = "abc" x = 'abc' x = ''' multi line '''
- String methods
- Python3
5.4. Scrapy
- https://scrapy.org/doc/
scrapy runspider a.py -o r.json- Create a project
scrapy startproject tutorialcd tutorialand putwhatevername.pytotutorial/spidersscrapy crawl quotesscrapy crawl quotes -o r.jsonscrapy crawl quotes -o r.csv -t csv
- Modify
tutorial/settings.pyROBOTSTXT_OBEY = FalseUSER_AGENT = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'AUTOTHROTTLE_ENABLED = True
6. CSS
6.1. Display, Float, Position, box-sizing
6.1.1. Display
- block
- inline
widthheighthave no effectmarginonly can be set horizontally
- inline-block
widthheightcan be set- spaces between inline-block's can be eliminated by
font-size:0;
6.1.2. Float
Clear float to fix parent height less than total children height
<div class="clearfix"> <div style="">Div 1 doesn't float</div> <div style="float: left;">Div 2</div> </div>
/* Solution 1 */ .clearfix:after { content: " "; display:block; clear:both; height:0; } /* Solution 2 */ .clearfix { display:inline-block; width:100%; } /* Solution 2 */ .clearfix { overflow:hidden; }
6.1.3. Position
- static
- Default position. left, right, top, bottom have no effect
- relative
- is relative to default static position, use left, right, top and bottom
- Other elements adjacent to
position:relativeelement will not be adjusted to fit into any gap left by the element (treat the relative element as static)
- Other elements adjacent to
- fixed
position:fixedis relative to viewport- Its element is removed from the normal document flow, and no space is created for the element in the page layout
- It's positioned relative to the initial containing block established by the viewport, except when one of its ancestors has a
transform,perspective, orfilterproperty set to something other thannone, in which case that ancestor behaves as the containing block. Its final position is determined by the values oftop,right,bottom,left - In printed documents, the element is placed in the same position on every page
- (no term)
- sticky
- It is positioned according to the normal flow of the document, and then offset relative its nearest scrolling ancestor and containing block (nearest block-level ancestor) including table-related elements
- scrolling ancestors are
overflowishidden,scroll,autooroverlay - It is positioned based on the value of
top,right,bottomandleft
- absolute
postion:absoluteis positioned relative to the nearest positioned ancestor. If there's no positioned ancestors, document body is used- The showing area of
position:absoluteelements depend on the outer most parent's width/height and padding which is<body>
- The showing area of
- positioned element
- its
positionis anything exceptstatic- When elements are positioned, they can overlap other elements
- it creates a stacking context
- (no term)
- Stacking context
- (no term)
- CSS Position from MDN
6.1.4. box-sizing
box-sizing: border-box | padding-box | content-box (default);
6.2. Horizontal Center Absolute
The absolute element is positioned related to the first position:relative parent
Center that abosulte element in relate to that parent
Notice: any parents will not expand in width and height according to that absolute child element
.grandchild {
position: absolute;
margin-left: auto;
margin-right: auto;
left: 0;
right: 0;
}
6.3. Transform, Transition
- See CSS Animation
6.3.1. Transition
transition: /transition-property/ /transition-duration/ /transition-timing-function/ /transition-delay/div { width: 100px; height: 100px; background: red; transition: width 2s linear 3s, height 2s linear 3s, transform 2s linear 3s; }
- transition-property
- can be 'all'
- transition-timing-function
- default linear
- ease-in
- slow start
- ease-out
- slow end
- (no term)
- e.g. use ease-in for div:hover and ease-out in div
- transition-delay
- delay n seconds before the current transition starts.
6.3.2. Transform
div:hover { width: 300px; height: 300px; transform: rotate(180deg) translate(-20px, 0) scale(0.9, 2) skew(30deg, 20deg); } /* You can turn off all transform */ /* transform: none; */
- 2D transforms
- translate()
- translate(tx) is equivalent to translate(tx, ty) as ty is 0 by default. Default is translate(0,0)
- (no term)
rotate(angle)
- Angle
- rad
- One full circle is 2π radians which approximates to 6.2832rad. 1rad is 180/π degrees. Examples: 0rad, 1.0708rad, 6.2832rad
- grad
- Represents an angle in gradians. One full circle is 400grad. Examples: 0grad, 100grad, 38.8grad.
- turn
- Represents an angle in a number of turns. One full circle is
1turn. Examples: 0turn, 0.25turn, 1.2turn
- scale()
- e.g. scale(0.5, 0.5)
- (no term)
- scaleX() scaleY()
- skew()
- skew(20deg, 10deg)
- skewX() skewY()
- skewX(20deg)
- matrix()
- matrix(scaleX(),skewY(),skewX(),scaleY(),translateX(),translateY())
- 3D transforms
- tranlsate3d(x,y,z)
- translateX(x) translateY(y) translateZ(z)
- scale3d(x,y,z)
- scaleX(x) scaleY(y) scaleZ(z)
- x, y, z is a vector and x|y|z is between 0 and 1, and rotate according to that vector
- rotateX(angle) rotateY(angle) rotateZ(angle)
- https://meyerweb.com/eric/tools/matrix/
- d is by default 0 and must be a positive number.
- tranlsate3d(x,y,z)
- 3D transform properties
- transform-origin
- e.g. rotate around transform-origin https://css-tricks.com/almanac/properties/t/transform-origin/
- transform-style
- whether child elements should carry parent's transform property https://css-tricks.com/almanac/properties/t/transform-style/
- perspective
- default none https://css-tricks.com/almanac/properties/p/perspective/
- perspective-origin
- 50% 50%|initial|inheri https://css-tricks.com/almanac/properties/p/perspective-origin/
- backface-visibility
- visible|hidden|initial|inherit
- http://ds-overdesign.com/transform/matrix3d.html
6.4. Insert CSS File css:js insert
['//ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/themes/ui-darkness/jquery-ui.css', '/wp-content/themes/xxx/css/interstitial.css'].forEach(function (src) { var _css = document.createElement("link"); _css.rel = 'stylesheet'; _css.type = 'text/css'; _css.href = src; document.head.appendChild(_css); });
6.5. Font, Text css:font
6.5.1. Ellipsis
Single Line
div.test {
white-space: nowrap;
width: 200px; // width has to be defined
overflow: hidden; // needed
text-overflow: ellipsis; // change to inherit when hover then the clipped text will show
}
Multiple Lines (IE and Edge will not show 3 dots)
.excerpt { display: block; display: -webkit-box; font-weight: normal; font-size: 15px; line-height: 1.4; -webkit-line-clamp: 9; -webkit-box-orient: vertical; overflow: hidden; text-overflow: ellipsis; height: 189px; /* 15*1.4*9 */ }
6.5.2. Long Word
word-wrap: break-word; // allow to break a long word
6.5.3. Font size, line-height, letter-spacing
font-sizein child elements inherit parent elementemuses the current element's font-size or its closest parent element which hasfont-sizedefinedremis relative to the<html>'s font-size- e.g. set
remforline-heightin<body>to ensureline-heightis relative to <html>'s font-size
- e.g. set
- Both em and rem will be affected by parent element's css:zoom in Edge!
vwandvhare the viewport width and height. 1vw is 1% of vw- Google recommends
line-heightis at least 1.2line-heightis normal means to use user agent style. Most likely default is 1 not 1.2
letter-spacing, padding using em is relative to the current element's font size
6.5.4. Prevent sup and sub affecting line height css:sup:line-height
sup, sub {
vertical-align: baseline;
position: relative;
top: -0.4em;
}
sub {
top: 0.4em;
}
.sup {
line-height:0;
font-size:6px;
vertical-align:5px;
}
6.5.5. Fluid Typography
body {
font-size: calc([minimum size] + ([maximum size] - [minimum size]) * ((100vw - [minimum viewport width]) / ([maximum viewport width] - [minimum viewport width])));
}
// example
html {
font-size: 16px; /* [minimum size] */
}
/* 320px = [minimum viewport width] */
@media screen and (min-width: 320px) {
html {
font-size: calc(16px + (22 - 16) * ((100vw - 320px) / (1000 - 320) ) );
}
}
/* 1000px = [maximum viewport width] */
@media screen and (min-width: 1000px) {
html {
font-size: 22px; /* [maximum size] */
}
}
6.5.6. @font-face css:font-face
- Most widely accepted font format is WOFF2, WOFF then TTF and OTF
- On macOS
- the Font Book.app can only add TTF, TTC, and OTF
- If you find macOS renders thicker glyphs for light text on dark background thicker than other browsers, change:
-webkit-font-smoothingto beantialiasedfor Safari and Chrome-moz-osx-font-smoothingto begrayscalefor Firefox- Do nothing for Windows and Linux
-webkit-font-smoothing- auto
- Uses subpixel anti-aliasing when available; this is the default
- none
- Turn font smoothing off; display text with jagged sharp edges
- antialiased
- Smooth the font on the level of the pixel, as opposed to the subpixel. Switching from subpixel rendering to anti-aliasing for light text on dark backgrounds makes it look lighter.
- eq. to Firefox's grayscale
-moz-osx-font-smoothing: grayscale;
- eq. to Firefox's grayscale
- subpixel-antialiased
- On most non-retina displays, this will give the sharpest text.
-moz-osx-font-smoothing- auto
- Allow the browser to select an optimization for font smoothing, typically grayscale.
- grayscale
- Render text with grayscale anti-aliasing, as opposed to the subpixel. Switching from subpixel rendering to anti-aliasing for light text on dark backgrounds makes it look lighter.
*-font-smoothinghas no effect, so the default antialising mode issubpixel-antialiasing- In short, you need a different font source file for macOS
- On macOS
- IE 11 and less only supports EOT. Edge supports WOFF
- Use full url in
src: url() format()for email HTMLformat()- Optional. Can be: woff2, woff, truetype, opentype, embedded-opentype, svg
- For a Variable Font, can be a range e.g.
font-weight: 100 900;font-stretch: 25% 140%;
@font-face { font-family: myFirstFont; src: url(sansation_light.woff) format('woff'); } /* Another for bold */ @font-face { font-family: myFirstFont; src: url(sansation_bold.woff) format('woff'); font-weight: bold; /* font-stretch: normal | condensed | semi condensed | extra condensed | ultra condensed | expanded | ...; font-style: normal | italic | oblique */ }
6.5.6.1. font-display under font-face
- Chrome and Firefox will use the font fallback if the first font choice is not available after 3 seconds (timeout). If the first font choice is available later, a swap occurs
- IE has 0 timeout and use fallback immediately and swap later if font is available
- Safari always wait for the font to be downloaded
- Not supported in IE, Edge and iOS < 11.4
- auto
- use user-agent's default. Most user-agents use block
- block
- After 3 seconds if the font is not loaded use a invisible text and swap to the downloaded font with infinite swap period
- swap
- browser draws text immediately with a fallback and swap to it after it's loaded.
- fallback
- fallback at first and swap to it later. If too much time passes (3s), then the fallback is used for the rest of page lifetime.
- optional
- the font is a nice to have but not critical. It leaves it up to the browser to decide whether to initiate the font download
@font-face { font-family: 'Awesome Font'; font-style: normal; font-weight: 400; font-display: auto; /* or block, swap, fallback, optional */ src: local('Awesome Font'), url('/fonts/awesome-l.woff2') format('woff2'), /* will be preloaded */ url('/fonts/awesome-l.woff') format('woff'), url('/fonts/awesome-l.ttf') format('truetype'), url('/fonts/awesome-l.eot') format('embedded-opentype'); unicode-range: U+000-5FF; /* Latin glyphs */ }
6.5.6.2. Font Loading API
- https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/webfont-optimization
- https://medium.com/@matuzo/getting-started-with-css-font-loading-e24e7ffaa791
- Not supported in IE and Edge
var font = new FontFace("Awesome Font", "url(/fonts/awesome.woff2)", { style: 'normal', unicodeRange: 'U+000-5FF', weight: '400' }); // don't wait for the render tree, initiate an immediate fetch! font.load().then(function() { // apply the font (which may re-render text and cause a page reflow) // after the font has finished downloading document.fonts.add(font); document.body.style.fontFamily = "Awesome Font, serif"; // OR... by default the content is hidden, // and it's rendered after the font is available var content = document.getElementById("content"); content.style.visibility = "visible"; // OR... apply your own render strategy here... }); // check status :: unloaded, loading, loaded, error font.status // Promise loaded resolves when the font is loaded notoSansRegular.loaded.then((fontFace) => { // This is where you can declare a new font-family, because the font is now loaded and ready. console.info('Current status', fontFace.status); console.log(fontFace.family, 'loaded successfully.'); // Throw an error if loading wasn't successful }, (fontFace) => { console.error('Current status', notoSansRegular.status); }); // when all fonts are ready document.fonts.ready.then((fontFaceSet) => { console.log(document.fonts.size, 'FontFaces loaded.'); });
6.5.6.3. Bootstrap 3
@font-face {
font-family: 'Glyphicons Halflings';
src: url('../fonts/glyphicons-halflings-regular.eot');
src: url('../fonts/glyphicons-halflings-regular.eot?#iefix') format('embedded-opentype'),
url('../fonts/glyphicons-halflings-regular.woff2') format('woff2'),
url('../fonts/glyphicons-halflings-regular.woff') format('woff'),
url('../fonts/glyphicons-halflings-regular.ttf') format('truetype'),
url('../fonts/glyphicons-halflings-regular.svg#glyphicons_halflingsregular') format('svg');
}
6.5.6.4. Google Font and local() unicode-range
local()unicode-range
- Example
<link href="https://fonts.googleapis.com/css?family=Montserrat:400,700" rel="stylesheet" type="text/css"> - if user agent has the font, then the font will not be downloaded css:font-face:local
- Refer to html:link:rel:preload:font
- ranges are separated by comma
- Sigle codepoint
U+416- Interval range
- start and end e.g.
U+400-4ff - Wildcard range
?is any hexadecimal digit e.g.U+4??
- benefit of loading variants of the same font
- Can't make a bold font lighter
- Can't make an oblique (italic) font "straight"
- Has limited ability to synthesize bold(er) fonts
- Has limited ability to synthesize oblique fonts and may produce wrong shapes esepcially in Cyrillic fonts
/* cyrillic-ext */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 400; src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459WRhyzbi.woff2) format('woff2'); unicode-range: U+0460-052F, U+1C80-1C88, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F; } /* cyrillic */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 400; src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459W1hyzbi.woff2) format('woff2'); unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116; } /* vietnamese */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 400; src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459WZhyzbi.woff2) format('woff2'); unicode-range: U+0102-0103, U+0110-0111, U+1EA0-1EF9, U+20AB; } /* latin-ext */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 400; src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459Wdhyzbi.woff2) format('woff2'); unicode-range: U+0100-024F, U+0259, U+1E00-1EFF, U+2020, U+20A0-20AB, U+20AD-20CF, U+2113, U+2C60-2C7F, U+A720-A7FF; } /* latin */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 400; src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459Wlhyw.woff2) format('woff2'); unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; } /* cyrillic-ext */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 700; src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gTD_u50.woff2) format('woff2'); unicode-range: U+0460-052F, U+1C80-1C88, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F; } /* cyrillic */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 700; src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3g3D_u50.woff2) format('woff2'); unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116; } /* vietnamese */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 700; src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gbD_u50.woff2) format('woff2'); unicode-range: U+0102-0103, U+0110-0111, U+1EA0-1EF9, U+20AB; } /* latin-ext */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 700; src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gfD_u50.woff2) format('woff2'); unicode-range: U+0100-024F, U+0259, U+1E00-1EFF, U+2020, U+20A0-20AB, U+20AD-20CF, U+2113, U+2C60-2C7F, U+A720-A7FF; } /* latin */ @font-face { font-family: 'Montserrat'; font-style: normal; font-weight: 700; src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gnD_g.woff2) format('woff2'); unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD; }
- Example
- Google Fonts API - CSS
https://developers.google.com/fonts/docs/getting_started
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Tangerine">
- Replace any spaces in the font family name with plus signs (+)
- Styles and weights
- Styles
- italic
- or i
- bold
- or b or a numerical weight e.g. 700
- bolditalic
- or bi
- https://fonts.googleapis.com/css?family=Cantarell:italic%7CDroid+Serif:bold
- https://fonts.googleapis.com/css?family=Cantarell:i%7CDroid+Serif:b
- https://fonts.googleapis.com/css?family=Cantarell:i%7CDroid+Serif:700
- Styles
- Multiple font families
- 6.5.6.1
- Subset or script
- Just a handful of letters instead of loading all the letters
- no need to use
subset=xxx - https://fonts.googleapis.com/css?family=Inconsolata&text=Hello%20World
- no need to use
- Effect
- tackle variable fonts
- https://developers.google.com/fonts/docs/css2
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Crimson+Pro">- Multiple font families
- Google Web Font Loader - JavaScript
- https://github.com/typekit/webfontloader
- A JavaScript library to dynamic font loading Google and Typekit fonts
Synchronous load
<script src="https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js"></script> <script> WebFont.load({ google: { families: ['Droid Sans', 'Droid Serif', 'Roboto:300,400,700', ] } }); </script>
Async load
WebFontConfig = { google: { families: ['Poppins:400,500'] } }; (function(d) { if ( typeof WebFont === 'undefined' ) { var wf = d.createElement('script'), s = d.scripts[0]; wf.src = 'https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js'; wf.async = true; s.parentNode.insertBefore(wf, s); } else { WebFont.load(WebFontConfig); } })(document);
- https://github.com/typekit/webfontloader
6.5.6.5. Web Safe Fonts css:font-face:web safe fonts
- Arial, Arial Black
- Comic Sans MS
- Courier New
- Georgia
- Impact / Charcoal
- Lucida Sans / Lucida Grande
- Tahoma / Geneva
- Times New Roman
- Trebuchet MS
- Verdana
Web Safe Fonts in PC and Mac and font stack
6.5.7. font-feature-setting css:font-feature-settings
- Some fonts have OpenType Features. Refer to font:woff
/* Use the default settings */ font-feature-settings: normal; /* Set values for OpenType feature tags */ font-feature-settings: "smcp"; /* same as font-feature-settings: "smcp" 1; */ font-feature-settings: "smcp" on; /* same as font-feature-settings: "smcp" 1; */ font-feature-settings: "swsh" 2; font-feature-settings: "smcp", "swsh" 2; /* Global values */ font-feature-settings: inherit; font-feature-settings: initial; font-feature-settings: unset; /* Syntax normal | <string> [ <integer> | on | off ]? */
6.5.8. font-variation-settings css:font-variation-settings
Change value for each axis for a Variable Font
font-variation-settings: "wght" 370; font-variation-settings: "wght" 370, "wdth" 75, "YOPQ" 54;
Fallback
h1 { font-family: some-non-variable-font-family; } @supports (font-variation-settings: 'wdth' 115) { h1 { font-family: some-variable-font-family; } }
6.6. CSS Selectors CSS Selectors
div > p- select p elements where the direct parent is div
div + pselect p elements that are placed immediately after div
<div></div> <p></p>
p ~ ulselect ul elements that are preceded by a p element
<p></p> <ul></ul> <ul></ul>
a[target]- <a>'s with target attribute
a[target=_blank]
a[title~=flower]- containing a word or followed by a hypen (not match flowers)
a[lang|=en]- starting with a word or followed by a hyphen (en, en-us)
div[class^="test"]- starting with (anything)
div[class$="test"]- ending with (anything)
a[href*="w3schools"]- containing
div[id="c1"], div[id="c2"]- id should be c1 or c2
div[id^="c1"][lang^="en-"]- id starts with c1 AND lang starts with en-
- (no term)
:root:root { /* matches <html> */ /* perfect place to define custom properties to be used by var() function */ --example-name: 5px 24px; --blue: #007bff; }
- (no term)
a:active- (no term)
a:hovera:link- Unvisited and normal.
div:linkmay be possible when div has href attribute - (no term)
a:visited- (no term)
p:evenp:odd::*jQuery only*- (no term)
p:firstp:lastp::after<img>can't have pseudo element- (no term)
p::before- (no term)
p::first-letter- (no term)
p::first-line- (no term)
p::selectionp::lang(it)- language attribute
div::selection- the selected portion of an element (e.g. select text)
- (no term)
input[type=text]::placeholder- (no term)
input:checked- (no term)
input:enabled,input[type="text"]:disabledinput[type=radio]:default,input[type=checkbox]:default- default checked radio or checkbox
- (no term)
input:focus- (no term)
input:invalid,input:validinput:optional- without required attribute
input:required- with required attribute
p:empty- has no children including text nodes
div:not(p)- all children elements except <p>'s
div:not(.aClass)- all div elements except the ones with class aClass
p:eq(2)- jQuery only index at 2, starting from 0
p:gt(2)p:lt(2)- jQuery only index greater than 2
p:contains('abc')- jQuery only contains abc text
p:parent- jQuery only Select p elements which are parents (has some children or text node)
div:has(p[class=a])- jQuery only select div which has
p.a p:first-child- <p>'s are the first child of any elements
- (no term)
p:last-childp:nth-child(2)- start with 1
p:nth-child(7n-1)- n starts with 1
- (no term)
tr:nth-child(even)- (no term)
tr:nth-child(odd)- (no term)
p:nth-last-child(2)p:only-child- <p>'s are the only child of any elements
div p:first-of-type- <div> has 4 children <p>'s and first child is <a>. Select 1st of <p> children.
- (no term)
div p:last-of-typediv p:nth-of-type(2)- can use n like nth-child
- (no term)
div p:only-of-type#news:target- anchor name id=news matches the URL hash
- (no term)
ul > li:not(:last-child):after
6.7. Specificity
(a, b, c, d) start with (0, 0 , 0 ,0) a - inline add 1 b - each id add 1 c - class, pseudo-class and attribute add 1 d - element add 1
Pseudo-class :not is not considered as pesudo-class in specificity calculation.
But anything inside it counts.
6.8. Functions
6.8.1. attr(data-attr-1) css:f:attr
a::after { content: " (" attr(href) ")"; }
6.8.2. calc() css:f:calc
- No parenthesis is allowed inside calc()
- Only css:f:attr and calc() are allowed inside calc()
/* Always leave 50px on both sides for space and take all that is left for width */ div { width: calc(100% - 100px); /* calculate */ margin-left: 50px; } /* 3 columns minus gutter */ .item { width: calc(100%/3 - 10px/3); }
6.8.3. counter css:f:counter
body { counter-reset: section; } h1 { counter-reset: subsection; } h1::before { counter-increment: section, content: "Section " counter(section) ". "; } h2::before { counter-increment: subsection; content: counter(section) "." counter(subsection) " "; }
6.8.4. var(x) css:f:var
:root { --body-font-family: Arial; --amstel-wght: 370; --amstel-wdth: 75; } * { font-variable-settings: 'wght' var(--amstel-wght), 'wdth' var(--amstel-wdth); } body { font-family: var(--body-font-family); }
6.8.5. minmax(x,y) css:f:minmax
/* sets minimum row height to 100px, max height to auto */ grid-column-rows: minmax(100px, auto);
6.8.6. repeat(x,y) css:f:repeat
grid-template-columns: repeat(3, 20px [col-start]) 5%;
6.9. border-image
border-image: source slice width outset repeat|initial|inherit; border-image: none 100% 1 0 stretch
6.10. background css:background
- See 6.13
background: bg-color bg-image position/bg-size bg-repeat bg-origin bg-clip bg-attachment initial|inherit; background: url(img_flwr.gif) right bottom no-repeat, url(paper.gif) left top repeat;
6.10.1. background-position
background-position: right center; /* align right on X and align center on Y */ background-position: right bottom; background-position: right 100px bottom -100px; /* for right/left or top/bottom, +- pixels */ background-position: calc(50% - 100px) calc(50% + 100px); /* for center, calc has be used */
6.10.2. background-size
background-size: 20px auto; background-size: 50% auto; /* sets the width of the bg image in percent of the parent element. */ background-size: contain; /* bg image might not cover all content area */ background-size: cover; /* content area might not see the whole bg image */ /* Stretch on y-axis and repeat on x-axis */ background: url() center repeat-x; background-size: auto 100%;
6.10.3. background-clip, background-origin
background-clip: border-box; / default. largest area / padding-box, content-box
/ For background image background-origin: border-box; / default. largest area // padding-box, content-box
6.10.4. background-attachment
- default scroll
- bg is fixed with regard to the element itself and does not scroll with its contents (bg is attached to the element's border). Fixed to document window.
- fixed
- bg is fixed with regard to the viewport. Even if an element has a scrolling mechanism. Bg doesn't move with the element
- local
- bg is fixed with regard to the element's contents: if the element has a scrolling mechanism, the bg scrolls with the element's contents. Fixed to both viewport and document window.
6.10.5. Add dark layer on a background image
15% darker
/* Before */ background: url(/slide-03.jpg) no-repeat center center; /* After */ background: linear-gradient(to bottom, rgba(0, 0, 0, 0.15) 0%, rgba(0, 0, 0, 0.15) 100%), url('/slide-03.jpg') no-repeat center center;
6.10.6. Mobile background-file:///Users/lili/li/notes/fixed background-size:cover
Instead of
.cs-hero-background { background: url('...') no-repeat center bottom fixed; -webkit-background-size: cover; -moz-background-size: cover; -o-background-size: cover; background-size: cover; }
Do this, body:before might work in some situation
.cs-hero-background:before { content: ""; display: block; position: fixed; left: 0; top: 0; width: 100%; height: 100%; z-index: -10; background: url('...') no-repeat center bottom; -webkit-background-size: cover; -moz-background-size: cover; -o-background-size: cover; background-size: cover; }
6.11. List style
<'list-style-type'> || <'list-style-position'> || <'list-style-image'> Can't control the image size.
6.12. Color
6.12.1. CSS inherit color
You can grab the parent `color` css property and use it in any color properties in child element
<div class="parent"> <div class="child"> </div> </div> <style> .parent { color: red; background-color: blue; } .child { background-color: currentColor; /* take the parent color: red */ color: currentColor; /* Take the parent color: red */ } </style>
6.12.2. rgb(), rgba() CSS rgba
#f03 #ff0033 #FF0033 rgb(255, 0, 51) // no fraction all integers rgb(100%, 0%, 20%) // all % no integer #f030 // 0% opaque red #ff003300 // 0% opaque red #FF003388 // 50% opaque red rgba(255,0,0,0.7) // 70% opaque red
6.12.3. hsl(), hsla()
Hue :: integer, angle degree of the color circle. Red is 0, Green is 120, Blue is 240 Saturation :: percentage. 0% is grey Lightness :: 100% is white, 0% is black, 50% is normal
hsla(240, 100%, 50%, 0.05) // 5% opaque blue
6.12.4. Darker or lighter color
6.13. Gradient and repeat gradient
Both CSS gradient and CSS repeat gradient can be stacked in background-image
linear-gradient(45deg, blue, red); /* to bottom, to top to right, to left to bottom right, to top left */ /* blue at 0%, white at 80%, orange at 100% */ linear-gradient(to bottom, blue, white 80%, orange); /* evenly distributed */ linear-gradient(to right, red, orange, yellow, green, blue); /* Use it with background image */ background: linear-gradient(to right, rgba(255,255,255,0), rgba(255,255,255,1)) ,url(/bg.jpg); /* non evenly distributed :: 0-50% solid black, 50%-100% solid transparent */ background: linear-gradient(to right, black 0%, black 50%, transparent 50%, transparent 100%); /* evenly spaced */ radial-gradient(red, yellow, rgb(30, 144, 255)); /* non evenly distributed */ radial-gradient(red 5%, yellow 25%, #1E90FF 50%); /* With shape :: circle or ellipse Default farthest-sided Fade from the center point to the ___ closest-corner, closest-side, farthest-corner, farthest-side */ radial-gradient(circle, yellow, #f06d06); /* gradient from center point to corner */ radial-gradient( circle farthest-corner at top right, yellow, #f06d06 );
Repeat gradient
/ On top of percentage, px can be used in repeat gradient */ repeating-linear-gradient(-45deg, red, red 5px, white 5px, white 10px);
6.14. opacity
0, 0.3, 1. The lower the value the more transparent Child elements inherits the parent opacity value If you don't want to carry opacity to children, define rgba color in parent elements CSS rgba
6.15. Shadow
Shadows can be stacked
box-shadow: h-shadow v-shadow blur spread color |inset|initial|inherit; text-shadow: h-shadow v-shadow blur-radius color|none|initial|inherit;
Text stroke/outline
text-shadow: -1px -1px 0 #000, 0 -1px 0 #000, 1px -1px 0 #000, 1px 0 0 #000, 1px 1px 0 #000, 0 1px 0 #000, -1px 1px 0 #000, -1px 0 0 #000; // you might need to adjust font smoothing to antialiased font-smoothing: antialiased; -webkit-font-smoothing: antialiased; // there's a new way! Won't work on IE but work on all current browsers -webkit-text-fill-color:#ffd33a; -webkit-text-stroke: 1px #ffd33a;
6.16. object-fit
Apply this to <img>, <video> to have background effect. Usually width and height are set in those elements.
fill (default) | contain | cover | none | scale-down
<img src="..." class="cover">
img {
width: 150px;
height: 100px;
}
.cover {
-webkit-object-fit: cover;
-moz-object-fit: cover;
-ms-object-fit: cover;
-o-object-fit: cover;
object-fit: cover;
}
Edge and IE only works for <img> element
6.17. CSS target IE 11 and IE10
Target IE 11 only
_:-ms-fullscreen, :root .msie11 { color: blue; }
Target IE 10 and IE 11
@media all and (-ms-high-contrast:none) { .foo { color: green } /* IE10 */ *::-ms-backdrop, .foo { color: red } /* IE11 */ }
6.18. @media css:@media
6.18.1. Syntax
@media not|only /mediatype/ and (/media feature/) {}- html:meta:viewport
<link rel="stylesheet" media="mediatype and|not|only (media feature)" href="style.css">- css:@media:type
- Possible types
- all, print, screen, speech
- (no term)
not /mediatype/only /mediatype/- (no term)
They can be nested with
OR(separated by comma) but notand/* This works */ @media screen and (min-width:200px), not print and (min-width:300px) {} /* This syntax is wrong */ @media screen and (min-width:200px) and not print and (min-width:300px) {}
- (no term)
- Nested media queries (using commas) is supported on all browsers except IE11-
- Note
not /mediatype/is different fromnot (/mediafeature/)
- Media feature
- Each media feature,
key:valueorkey(Some media feature doesn't have a value) - must be wrapped in
()
- Each media feature,
6.18.2. Testing
var mql = window.matchMedia("(orientation: portrait)"); // "(max-width: 600px)" // window.addListener(handlerOrientationChange); handlerOrientationChange(mql); function handlerOrientationChange(mql) { if (mql.mathces) { console.log('media query matches'); } else { console.log('media query does not match'); } }
6.18.3. Features css:@media:features
min-width, max-width and:
| Key | Min/Max? | value |
| orientation | no | landscape, portrait. Not reliable when soft keyboard opens |
| color | yes | int, number of colors. (color) for color device |
| aspect-ratio | yes | int1/int2 |
| grid | no | 0, 1 or (grid). The last suggests it's a device with 1 font |
| width, height | yes | viewport |
| monochrome | yes | 0 (not mono) or int |
| resolution | yes | 300dpi or 2dppx |
| pointer | no | fine (mouse, drawying stylus), coarse (finger), none (not a pointing device) |
| hover | no | none (finger), hover (can have hover state) |
pointerandhoveralways refer to the primary device whileany-pointerandany-hoverrefer to any if existsany-hoverhason-demandwhich means mobile services can emulate hovering when the user performs a long tap- Don't use
min/max-device-widthas it refers to the screen size not the viewport size
6.18.4. Responsive Design
6.18.4.1. Mostly Fluid
<div class="container" role="main"> <div class="c1"> </div> <div class="c2"> </div> <div class="c3"> </div> <div class="c4"> </div> <div class="c5"> </div> </div>
.container { display: -webkit-flex; display: flex; -webkit-flex-flow: row wrap; flex-flow: row wrap; } .c1, .c2, .c3, .c4, .c5 { width: 100%; } @media (min-width: 600px) { .c2, .c3, .c4, .c5 { width: 50%; } } @media (min-width: 800px) { .c1 { width: 60%; } .c2 { width: 40%; } .c3, .c4, .c5 { width: 33.33%; } } @media (min-width: 800px) { .container { width: 800px; margin-left: auto; margin-right: auto; } }
6.18.4.2. Column Drop
<div class="container" role="main"> <div class="c1"></div> <div class="c2"></div> <div class="c3"></div> </div>
.container { display: -webkit-flex; display: flex; -webkit-flex-flow: row wrap; flex-flow: row wrap; } .c1, .c2, .c3 { width: 100%; } @media (min-width: 600px) { .c1 { width: 60%; -webkit-order: 2; order: 2; } .c2 { width: 40%; -webkit-order: 1; order: 1; } .c3 { width: 100%; -webkit-order: 3; order: 3; } } @media (min-width: 800px) { .c2 { width: 20%; } .c3 { width: 20%; } }
6.18.4.3. Layout shifter
<div class="container" role="main"> <div class="c1"></div> <div class="c4"> <div class="c2"></div> <div class="c3"></div> </div> </div>
.container { display: -webkit-flex; display: flex; -webkit-flex-flow: row wrap; flex-flow: row wrap; } .c1, .c2, .c3, .c4 { width: 100%; } @media (min-width: 600px) { .c1 { width: 25%; } .c4 { width: 75%; } } @media (min-width: 800px) { .container { width: 800px; margin-left: auto; margin-right: auto; } }
6.18.4.4. Off canvas
<div class="container" role="main"> <div class="c1" id="leftDrawer"> </div> <div class="c2" id="mainPanel"> Click the <code>div</code>'s to change views </div> <div class="c3" id="rightDrawer"> </div> </div>
body { overflow-x: hidden; } .container { display: block; } .c1, .c3 { position: absolute; width: 250px; height: 100%; /* This is a trick to improve performance on newer versions of Chrome #perfmatters */ -webkit-backface-visibility: hidden; backface-visibility: hidden; -webkit-transition: -webkit-transform 0.4s ease-out; transition: transform 0.4s ease-out; z-index: 1; } .c1 { /* Using translate3d as a trick to improve performance on older versions of Chrome See: http://aerotwist.com/blog/on-translate3d-and-layer-creation-hacks/ #perfmatters */ -webkit-transform: translate(-250px,0); transform: translate(-250px,0); } .c2 { width: 100%; position: absolute; } .c3 { left: 100%; } .c1.open { -webkit-transform: translate(0,0); transform: translate(0,0); } .c3.open { -webkit-transform: translate(-250px,0); transform: translate(-250px,0); } @media (min-width: 500px) { /* If the screen is wider then 500px, use Flexbox */ .container { display: -webkit-flex; display: flex; -webkit-flex-flow: row nowrap; flex-flow: row nowrap; } .c1 { position: relative; -webkit-transition: none 0s ease-out; transition: none 0s ease-out; -webkit-transform: translate(0,0); transform: translate(0,0); } .c2 { position: static; } } @media (min-width: 800px) { body { overflow-x: auto; } .c3 { position: relative; left: auto; -webkit-transition: none 0s ease-out; transition: none 0s ease-out; -webkit-transform: translate(0,0); transform: translate(0,0); } }
6.18.5. Viewport sizes of device
- Refer to tool:device metrics
// @media only screen and (max-width:1920) {} // @media all and (...) {...} @media (max-width:1920px){} /* iPadPro@H1366 */ @media (max-width:1400px){} /* iPad@H1024 */ @media (max-width:1267px){} /* iPadPro@1024 */ @media (max-width:1201px){} @media (max-width:1023px){} @media (max-width:967px){} /* iPad@768, iPhone6+@H736, Nexus 6P/5X@H732 */ @media (max-width:867px){} /* iPhone6@H667 */ @media (max-width:697px){} /* GalaxyS5@H640 */ @media (max-width:645px){} /* iPhone5@H568 */ @media (max-width:597px){} @media (max-width:467px){} /* iPhone6+@414, Nexus 6P/5X@412 */ @media (max-width:420px){} /* iPhone6@375, GalaxyS5@360 */ @media (max-width:375px){} /* iPhone5@320 */ @media (max-width:320px){}
Mobile first
.any { width:100%; } @media (min-width:576px) {} @media (min-width:768px) {} @media (min-width:992px) {} @media (min-width:1200px) {}
6.19. @keyframes
- IE10+, iOS 6.1+, Android 4.4.4+
animation: /animation-name/ /animation-duration/ /animation-timing-function/ /animation-delay/ /animation-iteration-count/ /animation-direction/ /animation-fill-mode/ /animation-play-state/;<i class="fa fa-repeat fa-spin"></i>@keyframes fa-spin { 0% { transform: rotate(0deg); } /* this is called a step */ 100% { transform: rotate(359deg); } /* 0% and 100% steps share the same CSS, you can */ /* 0%, 100% {} */ /* Some effects have high performance animation: transform: translate() transform: scale() transform: rotate() opacity */ } .fa-spin { animation: fa-spin 2s linear 3s infinite alternate; /* 2s :: animation-duration Xs or Xms 3s :: animation-delay Xs or Xms. delay before start, once started, the delay has no effect infinite :: animation-iteration-count inifinite or int X alternate :: animation-direction normal,alternate animation-play-state :: paused, running animation-fill-mode: none (default, after animation ends, styles go back to before start) | forwards (after animation ends, styles got applied) | backwards (before animation starts and during delay, apply styles defined in the 1 iteration, e.g. from and to ) | both; multiple animations can be separated by comma */ } .fa-repeat:before { content: "\f01e"; }
ease (default), ease-in, ease-out, ease-in-out, linear, step-start, step-end
.move-me { display: inline-block; padding: 20px; color: white; position: relative; margin: 0 0 10px 0; } /* jump-term :: jump-start/start, jump-end/end, jump-none, jump-both - start :: the first jump happens when the animation begins - end :: the last jump happens when the animation ends */ .move-me-1 { animation: move-in-steps 8s steps(4) infinite; } .move-me-2 { animation: move-in-steps 8s steps(4, start) infinite; } .move-me-3 { animation: move-in-steps 8s infinite; } body { padding: 20px; } @keyframes move-in-steps { 0% { left: 0; background: blue; } 100% { left: 100%; background: red; } }
Javascript starts keyframes animation
<div id="myDIV" onclick="startAnimation()">Click to start animation</div>var x = document.getElementById("myDIV"); function startAnimation() { x.style.animation = "fa-spin 2s liner 3s infinite alternate"; // Standard }
Javascript animation event listners
x.addEventListner("animationstart", func1); x.addEventListner("animationiteration", func2); x.addEventListner("animationend", func3);
- See Animate.css
6.20. @import
/* @import usage should be avoided and should appear before any css styles defined in a css file */ @import "mystyle.css"; @import url("mystyle.css"); @import "mystyle.css" print; /* only media types not media features */
6.21. @viewport
WD. Only Edge. Same as html:meta:viewport
@media screen and (max-width: 400px) { @-ms-viewport { width: 320px; } /* Define the viewport size */ }
6.22. @supports css:@supports
@supports not (display:flex) {} @supports (display: flex) or (display: -ms-flexbox) or (display: -moz-box) {} @supports (transition-property: color) or ( (animation-name: myAnimation) and (transform: rotate(225deg) ) ) {}
6.23. Flexbox
6.23.1. Basics
- Container could be
display:flexordisplay:inline-flex - Flexbox is good for aligning child elements with irregular or unknown or inconsistent width (x-axis default) and height and unknown number of child elements
- Good for vertical and horizontal center a child element inside a container (parent)
- https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Flexible_Box_Layout/Typical_Use_Cases_of_Flexbox
- Can't set equal width for child items when the number of child items is unknown
.container { display: flex; /* flex-flow: <flex-direction> <flex-wrap> */ /* flex-direction: row | row-reverse | column | column-reverse; */ /* main axis is by default horizontally left to right * main axis is changed by direcion css property (say change it ot rtl, right to left) * row is left to right , column is top to bottom * */ /* flex-wrap: nowrap | wrap | wrap-reverse; */ flex-flow: row wrap; /* justify-content: flex-start | flex-end | center | space-between | space-around; */ /* e.g. you can change from flex-end to space-around for menu nav with @media queries */ justify-content: space-around; /* align-items: stretch | flex-start | flex-end | center | baseline; This is for aligning items on one line (default x axis) on cross axis (y axis) relative to the same line. e.g. you have 2 lines of items and each line holds 3 items. 3 items on each line have different height how do the 3 items align on y axis on every line? default is stretch which means all 3 items on the same line have equal height You have 2 options to set the items to have intrinsic height 1. Set align-items: flex-start on the container, then every item on a line has a different height 2. Set align-self: flex-start on an item to make that item have its intrinsic height */ /* align-content: flex-start | flex-end | center | space-between | space-around | stretch; * This is for space between multiple lines * e.g. There're 3 lines of items and each line has 3 items * How do the 3 lines align on the y axis on each line? * default is stretch * It only works when there are multiple lines and the container height is larger than total children height */ /* It's always to set width and height in container */ width: 100%; } .item { /* order: <integer>; // optional */ /* flex: none | <flex-grow> <flex-shrink> <flex-basis> */ /* Default flex:0 1 auto; */ /* flex-grow: 0 | <number>; Determines how the flex-item will expand if there's extra space in the container 0 is default and means do not enlarge item in width (default x axis) */ /* flex-shrink: 1 | <number> Determines how the flex-item will shrink if there isn't enough space in the container 1 is default and means the item can be shrank in width (default x axis) Set to 0, flex-items might overflow */ /* flex-basis: auto | <length>; Sets the initial size of the flex-item if it's 0, the final width of the flex-item only depends on grow/shrink factor ratio. Using auto or others, the final width is based on that set length and the amount of grow/shrink Set to 0% instead of 0 */ /* don't set width in flex item */ flex: 0 0 100px; /* instead of width:100px */ flex: 0 0 50%; /* width: 50% */ /* However, flex-basis is not guaranteed and it's based on max-width and min-width */ flex: 0 0 100%; /* use this in child item instead of width:100%, make sure parent has flex-wrap: wrap */ flex: 1 0 auto; /* grow to the rest of the width, no shrink, auto */ /* Override align-items in container */ /* align-self: stretch | auto | flex-start | flex-end | center | baseline; */ } .item_1 { /* Set margin:auto in an item will absorb all space of the current line in current direction margin-right: auto; /* all other items to the right of item_1 will not have extra space */ }
6.23.2. Insert a line break in flex items. Might not be easy..
.line-break {width: 100%}
<div class="container">
<div class="item">1</div>
<div class="item">2</div>
<div class="line-break"></div>
<div class="item">3</div>
<div class="item">4</div>
</div>
6.23.3. flex-grow:1
You can flex-grow a child's width or height (depending on the flex-direction of the container), so that the child takes up the remaining width or height of the container. Just remember to set the width or height for the container.
Combine with Bootstrap div.container and div.row, you can set a specific row to take the rest of the container height. You don't have to get height of any other rows.
.container.flex-column { display:flex; flex-direction:column; height: 100%; /* Make sure the container has a height */ /* This is to take the height of the parent of the container */ } .row.cover-row { flex-grow:1; }
6.23.4. Examples
.flex-r-nw-sb { /* row nowrap space-between */ display:flex; flex-flow: row nowrap; width:100%; } .flex-r-nw-sb-fs { /* row nowrap space-between don't stretch items' height on each line */ display:flex; justify-content:space-between; flex-flow: row nowrap; align-items:flex-start; width:100%; } .flex-r-nw-sb-mq-r-w-c { /* row nowrap space-between */ /* same as flex-r-nw-sb but with @media query */ display:flex; flex-flow: row nowrap; width:100%; } @media only screen and (max-width: 597px) { .flex-r-nw-sb-mq-r-w-c { flex-flow: row wrap; } }
6.24. Equal width for unknown number of child elements
<div> <span>tab1</span> <span>tab2</span> <span>tabx</span> </div>
div{ margin:0; padding:0; width:100%; max-width:100%; display:table; /* trick */ table-layout: fixed; /* trick */ } span{ border:1px solid grey; padding 0 20; margin:0; overflow:hidden; text-overflow: ellipsis; white-space: nowrap; display:table-cell; /* trick */ } /* Set back div { display:block; table-layout:auto; } span { display:inline; } */
6.25. Grid
6.25.1. container: grid, grid-template
gridhttps://developer.mozilla.org/en-US/docs/Web/CSS/grid
grid: <'grid-template'> | <'grid-template-rows'> / [ auto-flow && dense? ] <'grid-auto-columns'>? | [ auto-flow && dense? ] <'grid-auto-rows'>? / <'grid-template-columns'>
grid-templatehttps://developer.mozilla.org/en-US/docs/Web/CSS/grid-template
.container { grid-template: none | subgrid | <grid-template-rows> / <grid-template-columns>; } /* Keyword value */ grid-template: none; /* grid-template-rows / grid-template-columns values */ grid-template: 100px 1fr / 50px 1fr; grid-template: auto 1fr / auto 1fr auto; grid-template: [linename] 100px / [columnname1] 30% [columnname2] 70%; grid-template: fit-content(100px) / fit-content(40%); /* grid-template-areas grid-template-rows / grid-template-column values */ grid-template: "a a a" "b b b"; grid-template: "a a a" 20% "b b b" auto; grid-template: [header-top] "a a a" [header-bottom] [main-top] "b b b" 1fr [main-bottom] / auto 1fr auto; /* Global values */ grid-template: inherit; grid-template: initial; grid-template: unset;
6.25.2. container: display, grid-template-columns, grid-template-rows
css:grid:container:grid-template-columns css:grid:container:grid-template-rows
- Can use css:f:minmax
<!-- div.container has to be the direct parent of all grid items --> <div class="container"> <!-- .item are direct descendants --> <div class="item item-1"></div> <div class="item item-2"> <div class="sub-item"> <!-- .sub-item is not a grid item! --> </div> </div> <div class="item item-3"></div> </div>
.container { display: grid | inline-grid | subgrid; /* subgrid to indicate the grid container is a grid item of another grid container and it should take the parent's sizes of rows/columns */ /* Don't use column, float, clear and vertical-align in grid container as they have no effect */ /* grid-template-columns: <track-size> ... | <line-name> <track-size> ...; grid-template-rows: <track-size> ... | <line-name> <track-size> ...; <track-size> - can be a length, a percentage, or a fraction of the free space in the grid (unit is fr) <line-name> - an arbitrary name */ grid-template-columns: 40px 50px auto 50px 40px; /* 5 columns */ grid-template-rows: 25% 100px auto; /* 3 rows */ /* Add line names grid-template-columns: [first] 40px [line2] 50px [line3] auto [col4-start] 50px [five] 40px [end]; grid-template-rows: [row1-start] 25% [row1-end row2-start] 100px [third-line] auto [last-line]; /* Any row or column line can have multiple names: [row1-end row-start] Any row or column line can have the same name */ grid-template-columns: repeat(3, 20px [col-start]) 5%; /* eq. to */ grid-template-columns: 20px [col-start] 20px [col-start] 20px [col-start] 5%; /* Free space is defined as total space minus any fixed size items */ grid-template-columns: 1fr 50px 1fr 1fr; /* each column will take (total width - 50px) * (1/3) */ }
6.25.3. container: grid-template-areas css:grid:container:grid-template-areas
- Refer to css:grid:item:grid-area
.item-a{ grid-area: header; } .item-b{ grid-area: main; } .item-c{ grid-area: sidebar; } .item-d{ grid-area: footer; } .container{ grid-template-columns: 50px 50px 50px 50px; grid-template-rows: auto; grid-template-areas: "header header header header" "main main . sidebar" "footer footer footer footer"; } /* . means that single cell is empty. You can have multiple dots without space ".." */ /* The first row line and column line of footer area have an extra name "footer-start" The last row line and column line of footer aree have an extra name "footer-end" */
6.25.4. container: grid-column-gap, grid-row-gap, grid-gap css:grid:container:grid-gap
Gutter size: space inbetween 2 row lines or column lines. But start and end space is always 0
grid-gap: <grid-row-gap> <grid-column-gap>; grid-column-gap: 10px; grid-row-gap: 15px;
6.25.5. container: justify-items, item: justify-self css:grid:container:justify-items
justify-items: start | end | center | stretch (default); /* Adjust content in a grid item along the row axis start: align to left end of the grid area end: right stretch: fills the whole width of the grid area */ .item { justify-self: /* To overwrite .container: justify-items */ }
6.25.6. container: align-items, item: align-self css:grid:container:align-items
align-items: start | end | center | stretch (default); /* Adjust content in a grid item along the column axis start: align content to the top of the grid area end: bottom stretch: fills the whole height of the grid area */ .item { align-self: start | end | center | stretch (default); }
6.25.7. container: justify-content css:grid:container:justify-content
Align the grid along the row axis. If all grid items are sized with non-flexible units like px, the total size of grid might be less than the size of its grid container. Total width of all grids is less than width of its grid container. start: align all grids to the left of the grid container end: right stretch: resize the grid items to allow the grid to fill the full width of the grid container space-around: place an even amount of space between each grid item, with half-sized spaces on the far ends space-between: place an even amount of space between each grid item, with no space at the far ends space-evenly: place an even amount of space between each grid item, including the far ends
justify-content: start | end | center | stretch | space-around | space-between | space-evenly
6.25.8. container: align-content css:grid:container:align-content
Align the grid along the column axis. Total height of all grids is less than height of its grid container. start: align all grids to the top of the grid container
align-content: start | end | center | stretch | space-around | space-between | space-evenly
6.25.9. container: grid-auto-columns, grid-auto-rows
css:grid:container:grid-auto-columns css:grid:container:grid-auto-rows
- Implicit grid tracks are created when items have positioned or resized (css:grid:item:grid-column, grid-row) outside of the defined grid (css:grid:container:grid-template-areas)
- Implicit grid tracks will have 0 width by default until it's set
- Implicit grid tracks of 2 columns and 2 rows are set
grid-template-columns: 60px 60px; grid-tempalte-rows: 90px 90px; /* 2x2 grids are created. 3 row lines and 3 column lines */ /* but item-b is out of bound */ .item-b { grid-column: 5 / 6; /* From the 5th column line to the 6th */ grid-row: 2 / 3; /* From the 2nd row line to the 3rd */ } /* It's 5x2 grids now. */ grid-auto-columns: 60px; /* the implicit grid tracks width is 60px but .item-b is auto */ grid-auto-rows: auto;
6.25.10. container: grid-auto-flow css:grid-auto-flow
For items that are not positioned (css:grid:item:grid-column, grid-row) nor put in a named area (css:grid:item:grid-area) which is later used by css:grid:container:grid-template-areas, these items are "not explicitly placed on the grid". Use this to control the auto placement
grid-auto-flow: row | column | row dense | column dense; /* row : fill rows first (default) column: fill columns first dense : fill smaller items first (order of items might change!) */ .container { /* 5x2 */ display: grid; grid-template-columns: 60px 60px 60px 60px 60px; grid-template-rows: 30px 30px; grid-auto-flow: row; } .item-a { /* 2x1 */ grid-column: 1; grid-row: 1 / 3; } .item-e { /* 2x1 */ grid-column: 5; grid-row: 1 / 3; }
<section class="container"> <div class="item-a">item-a</div> <div class="item-b">item-b</div> <div class="item-c">item-c</div> <div class="item-d">item-d</div> <div class="item-e">item-e</div> </section>
6.25.11. item: grid-column, grid-row css:grid:item:grid-column, grid-row
css:grid:item:grid-column-start css:grid:item:grid-column-end css:grid:item:grid-row-start css:grid:item:grid-row-end
Position and resize an item
grid-column: <grid-column-start> / <grid-column-end>; grid-row: <grid-row-start> / <grid-row-end>; .item-c { grid-column: 3 / span 2; grid-row: third-line / 4; }
6.25.11.1. item: grid-column-start, grid-column-end, grid-row-start, grid-row-end
css:grid:item:grid-column-start css:grid:item:grid-column-end css:grid:item:grid-row-start css:grid:item:grid-row-end
.item { grid-column-start: <number> | <name> | span <number> | span <name> | auto grid-column-end: <number> | <name> | span <number> | span <name> | auto grid-row-start: <number> | <name> | span <number> | span <name> | auto grid-row-end: <number> | <name> | span <number> | span <name> | auto } .item-a { grid-column-start: 2; grid-column-end: five; /* line-name */ grid-row-start: row1-start; grid-row-end: 3; } .item-b { grid-column-start: 1; grid-column-end: span col4-start; /* span to a line-name */ grid-row-start: 2; grid-row-end: span 2; /* span 2 rows */ }
6.25.12. item: grid-area css:grid:item:grid-area
Refer to css:grid:container:grid-template-areas
grid-area: <name> | <grid-row-start> / <grid-column-start> / <grid-row-end> / <grid-column-end>
6.25.13. item: justify-self, align-self css:grid:item:justify-self css:grid:item:align-self
Used to overwrite grid container's css:grid:container:justify-items, css:grid:container:align-items
6.25.14. IE10, IE11
IE doesn't support: https://rachelandrew.co.uk/archives/2016/11/26/should-i-try-to-use-the-ie-implementation-of-css-grid-layout/
- auto-placement (for each item, need to specify -ms-grid-row, -ms-grid-column, -ms-grid-row-span, -ms-grid-column-span)
- css:grid:container:grid-template-areas
display: -ms-grid; display: grid; grid-gap:10px; /* not supported in IE, use grid*/ -ms-grid-columns: 100px 10px 100px 10px 100px; /* 2 extra columns are for gap */ -ms-grid-rows: 100px 10px 100px; /* the extra row is for gap */ grid-template-columns: 100px 100px 100px; /* notice there are 3 columns here while in IE there are 5 columns */
.item
-ms-grid-row: 1; -ms-grid-column: 1; -ms-grid-column-span: 3; grid-column: 1 / 3; grid-row: 1;
6.26. Horizontal and Vertical Center
If both the parent and child are blocks, use flexbox.
For text or to place an absolute child element inside parent and the child element is not the only child. Refer to https://www.smashingmagazine.com/2013/08/absolute-horizontal-vertical-centering-css/
.center { height:200px; position:relative; } .center p { margin:0; position:absolute; top:50%; left:50%; transform: translate(-50%, -50%); }
bs:Center Tags Vertically center span inside h1
Vertically align <img> and <span> inside <a>
a { line-height:20px; display:block; vertical-align:middle; } a img { height: 15px; /* let's say the img's height is 15px */ } a span { display:inline-block; vertical-align:middle; line-height:1em; }
6.27. Column for text
<div> <h2>Title takes 2 columsn</h2> long text.. </div>
div { column-count: 3; column-gap: 40px; column-rule-style: solid; /* just like border */ column-rule-width: 1px; column-rule-color: lightblue; column-rule: 1px solid lightblue; } div h2 { column-span: 2; } div.fixedWidth { columns: 200px; /* column-count will be auto */ overflow-x: scroll; height: 600px; /* So that scrollbar shows on x axis */ }
6.28. Shape
At the end of 2016, only Chrome, Safari, iOS and Android browswers support shape.
<div class="conainer">
<img id="image" src="a.jpg">
<p id="content">Long text</p>
</div>
img {
display: block;
float:left;
width: 100%;
height: auto;
shape-outside: url(shape.png); /* polygon can be used */
/* png has areas that are partially or completely transparent */
shape-image-threshold: 0.9; /* The shape is any areas that have opacity greater than 0.9 */
shape-margin: 20px;
}
6.29. SVG
xml header should be removed when used in HTML
<?xml version="1.0" encoding="utf-8"?>
6.29.1. Make SVG, Optimize SVG, IE 11
Exported from AI and optimize it or use nodejs:svgo
IE 11 doesn't support url('a.svg') in background CSS. Use nodejs:svgo to convert to URL encoded.
div.svgbg {
background-image:url('url-encoded-svg');
}
6.29.2. SVG as background image IE 11 svg:ie11
AI export .svg by default is responsive and no width and height attributes are in <svg>
You may add width and height to <svg> based on viewBox
<!-- before --> <svg viewBox="0 0 22.77 28.96"> <!-- after --> <svg viewBox="0 0 22.77 28.96" width="22.77" height="28.96">
If it still doesn't work, move <styles> to attributes
- You need to export .svg in Adobe Illustrator using one of these mode for the Styling
- Presentation Attributes
fill="none" stroke="#cfdf00" stroke-width="0.44"- Inline Style
style="fill:none;stroke:#cfdf00;stroke-width:0.4399999976158142px"- (no term)
- But not Internal Style!
- If AI already exports the .svg, use nodejs:svgo:move styles
6.29.3. SVG as background image dynamic size
You can only control size in this method
.logo { background: url(logo.svg) no-repeat top left; background-size: contain; width: 100px; height: 82px; }
6.29.4. CSS in SVG
Inline CSS
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <style type="text/css" > <![CDATA[ circle.myGreen { stroke: #006600; fill: #00cc00; } circle.myRed { stroke: #660000; fill: #cc0000; } ]]> </style> <circle class="myGreen" cx="40" cy="40" r="24"/> <circle class="myRed" cx="40" cy="100" r="24"/> </svg>
External CSS
<?xml-stylesheet type="text/css" href="svg-stylesheet.css" ?> <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <circle cx="40" cy="40" r="24" style="stroke:#006600; fill:#00cc00"/> </svg>
Embedded CSS in SVG only affects the SVG not other elements in the global HTML if svg is not inline with global HTML.
Closed strokes forms shapes or path. Shapes or path's can be filled. And stroke (color) and stroke-width (default 1px when stroke is set) can be set.
6.29.5. SVG Zoom and Pan
<g id="parentGraphic"> <rect /> <text></text> </g> <!-- use copy an element --> <use href="#parentGraphic" transform="translate(40, 30) scale(0.9)"></use>
<!-- Original --> <svg currentScale="1" width="300px" height="200px" viewbox="0 0 300 200"> <!-- Don't use currentScale to zoom --> <!-- Move down 200/200*25=25px and move right 300/300=50 px --> <svg width="300px" height="200px" viewbox="-50 -25 300 200"> <!-- Enlarge 50% --> <svg width="300px" height="200px" viewbox="0 0 150 100"> <!-- Enlarge 50%, move down 200/100*25=50px and move right 300/150*50=100px --> <svg width="300px" height="200px" viewbox="-50 -25 150 100"> viewBox = <min-x> <min-y> <w> <h>
Assuming min-x and min-y is both 0 Original width and height is width and height (200x200)
50% smaller, just change w and h to w*2 and h*2
Now the circle radius is shrank from 100 to 50 You need to move down and right 50px
200/(w*2)*<min-y>=50 <min-y> = 50/200*(w*2)
<min-y> = -(target move down amount)/(svg width)*(w*(shrink factor)) <min-x> = -(target move right amount )/(svg height)*(h*(shrink factor))
6.29.6. SVG Sprite
- Use
<symbol>, and they don't display until you<use>them
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="beaker" viewBox="214.7 0 182.6 792"> <!-- add more accessibility --> <title>original-file's-title</title> <desc>original-file's-desc</desc> <!-- <path>s and whatever other shapes in here --> </symbol> <symbol id="shape-icon-2" viewBox="0 26 100 48"> <!-- <path>s and whatever other shapes in here --> </symbol> </svg> <!-- We ain't even need viewBox round here. --> <!-- xlink:href="https://mysite.ca/a.svg#shape-icon-1"--> <!-- xlink:href="a.svg#shape-icon-1"--> <!-- Or the <svg><symbol></symbol></svg> is in the same document, use the following --> <svg class="icon"> <use xlink:href="#shape-icon-1" /> </svg> <svg class="icon"> <use xlink:href="#shape-icon-2" /> </svg>
- About CSS styling for
<svg><use> - https://tympanus.net/codrops/2015/07/16/styling-svg-use-content-css/
- (no term)
- Polyfil svg4everybody searches and replaces
svg usewith<object>
New way using <symbol> only works when <svg><use> (inline). If you want SVG-as-<img> or SVG-as-background-image, use this:
<defs> <style> g { display: none; } g:target { display: inline; } </style> </defs> <g id="icon-clock">...</g> <g id="icon-heart">...</g> <g id="icon-arrow-right">...</g>
<div class="lili-svg-as-bg"></div> <style> .lili-svg-as-bg { background:url('//y.ca/a.svg#icon-clock') left top repeat; width:100%; height:400px; font-size:1rem; } </style>
You can also place each svg one after another
<view id="icon-clock-view" viewBox="0 0 32 32" /> <view id="icon-heart-view" viewBox="0 32 32 32" /> <view id="icon-arrow-right-view" viewBox="0 64 32 32" /> <img src="sprite.svg#svgView(viewBox(0, 0, 32, 32))" alt=""> // or <img src="sprite.svg#icon-heart-view" alt=""> <style> .icon-clock { background: url("sprite.svg#svgView(viewBox(0, 0, 32, 32))") no-repeat; // or background: url(sprite.svg#icon-clock-view) no-repeat; // or background: url("sprite.svg") no-repeat; background-size: 32px 96px; background-position: 0 -32px; } </style>
6.29.6.1. Old way
Old Way uses <defs>, viewBox has to be defined in <svg>
<svg> <defs> <g id="shape-icon-1"> <!-- all the paths and shapes and whatnot for this icon --> <g> <g id="shape-icon-2"> <!-- all the paths and shapes and whatnot for this icon --> <g> </defs> </svg> <!-- These viewBox's better be right or the icons won't look right! --> <svg class="icon" viewBox="214.7 0 182.6 792"> <use xlink:href="#shape-icon-1" /> </svg> <svg class="icon" viewBox="0 26 100 48"> <use xlink:href="#shape-icon-2" /> </svg>
6.30. Sass css:scss css:sass
6.30.1. Variables
$childMargin: 20px; $numItems: 4 !default; $enable-rounded: true !default; $border-radius: .25rem !default; $badge-border-radius: $border-radius !default; $t_min_width: 992px; $t_max_width: 3000px; $t_min_font: 46.29px; $t_max_font: ($t_max_width * $t_min_font / $t_min_width); // 123px .featured-media { display:block; float:left; font-size: #{3 * 16}px; // eq. font-size: (3 * 16px); font-size: $t_max_font; margin:0 #{$childMargin} 20px 0; width: calc( ( 100% - ( #{$numItems} - 1 ) * #{$childMargin} ) / #{$numItems} ); &.every_fourth{ margin-right:0; } } @include media-breakpoint-down(sm) { $numItems: 2; .featured-media { width: calc( ( 100% - ( #{$numItems} - 1 ) * #{$childMargin} ) / #{$numItems} ); &.every_second{ margin-right:0; } } }
6.30.2. &
a { &:hover { } &.myclass { /* a.myclass */ } } .entry { .single &, .page & { /* .single .entry, .page .entry */ /* Useful for multiple parents */ } &--col-1 { /* .entry--col-1 */ } }
6.30.3. @extend % sass:@extend
%message-shared { border: 1px solid #ccc; padding: 10px; color: #333; } .message { @extend %message-shared; } .success { @extend %message-shared; border-color: green; } /* css */ .success, .message { border: 1px solid #ccc; padding: 10px; color: #333; } .success { border-color: green; }
6.30.4. Functions
map-keys(("foo": 1, "bar": 2)) => "foo", "bar"map-get(("foo": 1, "bar": 2), "foo") => 1
6.30.5. Mixin @mixin @include
@mixin flex { // write the css here display: -webkit-flex; display: flex; } .row { @include flex; } @mixin grid($flex) { @if $flex { @include flex; } @else { display: block; } } @include grid(true); @mixin grid($flex, $full-width) { // multiple variables } @mixin grid($max-width, $flex: true) { // var with default have to be the end } @mixin padding($values...) { // noticie 3 dots @each $var in $values { padding: #{$var}; } } a { @include padding(2px 4px 6px); } @mixin padding($values) { @each $var in $values { padding: #{$var}; } } // without 3 dots, it will result: a { padding: 2px; padding: 4px; padding: 6px; } $style1: 100%, 70px, #fo6d06; $style2: (background: #bada55, width: 100%, height: 100px); @mixin box($width, $height, $background) { width: $width; height: $height; background-color: $background; } .fogdog { @include box($style1...); } .badass { @include box($style2...); } // result: .fogdog { width: 100%; height: 70px; background-color: #fo6d06; } .badass { width: 100%; height: 100px; background-color: #bada55; } // use @content @mixin small() { @media only screen and (max-width: 320px) { @content; } } @include small { // css code for small screens go here width:123px; } // result @media only screen and (max-width: 320px) { width:123px; }
6.30.6. Bootstrap
6.30.6.1. Variables
$grid-breakpoints: ( xs: 0, sm: 576px, md: 768px, lg: 992px, xl: 1200px ) !default; $container-max-widths: ( sm: 540px, md: 720px, lg: 960px, xl: 1140px ) !default; $grid-gutter-width: 30px !default; .myclass { $tempWidth: (map-get($container-max-widths,xl) - $grid-gutter-width )/12*6 - ($featuredSliderPadding+1)*2; width: $tempWidth/6 - $grid-gutter-width/2; }
6.30.6.2. Functions
// Returns a blank string if smallest breakpoint, otherwise returns the name with a dash in front. // Useful for making responsive utilities. // // >> breakpoint-infix(xs, (xs: 0, sm: 576px, md: 768px, lg: 992px, xl: 1200px)) // "" (Returns a blank string) // >> breakpoint-infix(sm, (xs: 0, sm: 576px, md: 768px, lg: 992px, xl: 1200px)) // "-sm" @function breakpoint-infix($name, $breakpoints: $grid-breakpoints) { @return if(breakpoint-min($name, $breakpoints) == null, "", "-#{$name}"); } // Minimum breakpoint width. Null for the smallest (first) breakpoint. // // >> breakpoint-min(sm, (xs: 0, sm: 576px, md: 768px, lg: 992px, xl: 1200px)) // 576px // sample usage $min: breakpoint-min(sm, $breakpoints) @function breakpoint-min($name, $breakpoints: $grid-breakpoints) { $min: map-get($breakpoints, $name); @return if($min != 0, $min, null); }
6.30.6.3. Mixins
- media-breakpoint-up
@mixin media-breakpoint-up($name, $breakpoints: $grid-breakpoints) { $min: breakpoint-min($name, $breakpoints); @if $min { @media (min-width: $min) { @content; } } @else { @content; } }
@each $breakpoint in map-keys($grid-breakpoints) { @include media-breakpoint-up($breakpoint) { $infix: breakpoint-infix($breakpoint, $grid-breakpoints); .flex#{$infix}-row { flex-direction: row !important; } .justify-content#{$infix}-start { justify-content: flex-start !important; } } } @include media-breakpoint-up(sm) { // ... } @include media-breakpoint-down(sm) {} @include media-breakpoint-between(sm,md) {}
6.31. Use Cases
6.31.1. iOS Hover
Other than <a> tag, :hover doesn't work on iOS. Try one of these workaround
// add onclick
<div onClick="return true" class="tast test">
Test
</div>
// or add cursor:pointer
.test {
background-color:blue;
}
div.test {
cursor:pointer;
}
div.test:hover {
background-color:red;
}
6.31.2. Image inside a div
Div's height is greater than img. add display:block in <img>
<div class="item"> <img src="http://placehold.it/230x20/?text=6" style="display:block"> </div>
6.31.3. Transparent Image
<img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" width="100" height="100">
6.31.4. Image scaling
IE 10+ and Edge removes any -ms-* CSS filters, so this does not work any more
img { -ms-interpolation-mode: bicubic; }
Use this plugin https://github.com/sukhoi1/ie-bicubic-img-interpolation-plugin
$( document ).ready(function() { if (navigator.appName == 'Microsoft Internet Explorer' || !!(navigator.userAgent.match(/Trident/) || navigator.userAgent.match(/rv:11/)) || (typeof $.browser !== "undefined" && $.browser.msie == 1)) { $('img.render').bicubicImgInterpolation({ crossOrigin: 'anonymous' //for demo purpose }); } });
6.31.5. Button Hover Double Quotes
<button class="button" style=""><span>Hover </span></button>
.button {
display: inline-block;
border-radius: 4px;
background-color: #f4511e;
border: none;
color: #FFFFFF;
text-align: center;
font-size: 28px;
padding: 20px;
width: 200px;
transition: all 0.5s;
cursor: pointer;
margin: 5px;
}
.button span {
cursor: pointer;
position: absolute;
opacity: 0;
top: 0;
right: -20px;
transition: 0.5s;
}
.button:hover span {
padding-right: 25px;
/*
push button text to left and create padding
for double quotes to push to the right of padding
*/
}
.button:hover span:after {
opacity: 1;
right: 0;
}
6.31.6. Text Label
Flair is a fixed font-size squared tag.
<span class="li-tag li-tag-color">Flair</span> .li-tag{ display:inline-block; margin-right: .5em; padding: 0 2px; border:1px solid; border-radius: 2px; font-size:x-small; white-space:nowrap; vertical-align:middle; } .li-tag-color{ background-color:#f5f5f5; color: #555; border-color: #ddd; } .li-tag-red{ background-color:#d9534f; color: #ffffff; border-color: #d9534f; }
Vertical align inside h1, h2 h1 .li-tag, h2 .li-tag, h3 .li-tag { margin-top: -0.5em; }
h1, h2 must have font-size defined
6.31.7. Button with icon on the left
// No space in button
<button class="li-button has-icon"><span><a href="">Youtube</a></span></button>
<style>
.li-button {
display:inline-block;
padding:0 8px 0 5.5px;
border:solid 1px transparent;
border-radius:.25em;
background-color:#0072BC;
color:#ffffff;
vertical-align:middle;
/* Overwrite default button styles */
outline:0;
text-decoration:none;
cursor:none;
}
.li-button.has-icon::before {
margin-right:6px;
background:url(icon.svg) no-repeat;
background-size:16px 12px;
width:16px;
height:12px;
content:'';
display:inline-block;
vertical-align:middle;
}
.li-button span {
vertical-align:middle;
font-size:12px;
}
</style>
6.31.8. Vertical Line Separator in <li>
<ul class="ul">
<li><a href="">1</a></li>
<li><a href="">2</a></li>
</ul>
.ul li:last-child {
margin-right: 5px;
}
.ul li:not(:first-child):before {
content: "|";
padding:0 5px;
}
6.31.9. Wrap h1 around a div on the right, float right sequence
<div class="div">
<a href=""><img></a>
</div>
<h1>Long text...</h1>
.div {
display:block;
float:right;
}
h1 {
clear: none; /* default is none */
}
/* Make it take the whole width */
@media (max-width: 425px) {
.div {
display:block;
text-align: center;
width:100%;
}
}
<div id="div1" style="float:right"></div> <div id="div2" style="float:right"></div> <h1>...</h1> <!-- Sequence: h1, div2, div1 -->
6.31.10. 100% width and with margin
The trick is not specify 100% width and set the margin Or
margin-left: 15px; /* width: 100% */ width calc(100% - 15px);
6.31.11. Align text with elements
Use flex
<div class="flex-r-w-fs-center">
<div class="app-information-title p-r-10 p-t-5"><span>A Title</span></div>
<div class="app-information-btns">
<a href="#" role="button" class="btn btn-green-overnight" target="_blank">Sign up</a>
<a href="#" role="button" class="btn btn-green-overnight">FAQs</a>
<a href="#" role="button" class="btn btn-green-overnight" target="_blank">Learn more</a>
</div>
</div>
.app-information-title {
font-size:26px;
font-family: 'tg-medium';
font-weight: normal;
}
.app-information-title span{
line-height:33px;
}
6.31.12. Circle Number List
.numberCircle {
border-radius: 50%;
behavior: url(PIE.htc); /* remove if you don't care about IE8 */
width: 36px;
height: 36px;
padding: 8px;
background: #fff;
border: 2px solid #666;
color: #666;
text-align: center;
font: 32px Arial, sans-serif; /* 32 + padding/2 = width = height */
}
<span style="font-size: 450%; color: #FF5722;">①</span>
<span class="step">1</span>
span.step {
background: #cccccc;
border-radius: 0.8em;
-moz-border-radius: 0.8em;
-webkit-border-radius: 0.8em;
color: #ffffff;
display: inline-block;
font-weight: bold;
line-height: 1.6em;
margin-right: 5px;
text-align: center;
width: 1.6em;
}
Just change the font size to enlarge/shrink the button. Or width, line-height and the border-radius to only affect the circle (border-radius = half of the width and line-height values)
6.31.13. superscript, subscript uneven line height
<p>Some Text <sup>OM</sup>Office Mark
p {
line-height:1.5em; /* if line-height:0 doesn't work, try increase p line-height or lower sup font-size*/
}
sup {
/* font-size:0.5em; */
line-height:0;
}
6.31.14. Swap image in @media
css
.flex-r-w-fs-center-mq-r-w-sa-center { display:flex; flex-flow:row wrap; justify-content:flex-start; align-items:center; } .footer-logos > div { margin:15px 10px; background-color:red; } .footer-logos > div > div > img{ display:block; margin:0 auto; } @media only screen and (max-width: 697px) { .footer-logos.flex-r-w-fs-center-mq-r-w-sa-center { justify-content:space-around; } } @media only screen and (max-width: 320px) { .footer-logos > div { width:100%; margin-top:2em; } .footer-logos .cs-i-1 { background:transparent url('http://placehold.it/137x42/?text=137x42') center center no-repeat; height:42px; } .footer-logos .cs-i-2 { background:transparent url('http://placehold.it/137x49/?text=137x49') center center no-repeat; height:49px; } .footer-logos .cs-i-3 { background:transparent url('http://placehold.it/137x64/?text=137x64') center center no-repeat; height:64px; } .footer-logos .cs-i-4 { background:transparent url('http://placehold.it/137x37/?text=137x37') center center no-repeat; height:37px; } .footer-logos .cs-items { margin:0.5em auto; width:137px; } .footer-logos .cs-items img { visibility:hidden; } }
<div class="footer-logos flex-r-w-fs-center-mq-r-w-sa-center"> <div> <div class="cs-items cs-i-1"> <img src="http://placehold.it/81x25/?text=81x25" alt="#1 Logo"/> </div> </div> <div> <div class="cs-items cs-i-2"> <img src="http://placehold.it/92x25/?text=92x25" alt="#2 Logo"/> </div> </div> <div> <div class="cs-items cs-i-3"> <img src="http://placehold.it/54x25/?text=86x25" alt="#3 Hydro"/> </div> </div> <div> <div class="cs-items cs-i-4"> <img src="http://placehold.it/70x25/?text=70x25" alt="#4 Logo"/> </div> </div> </div>
6.31.15. Hightlight element when the hash and id of element match
:target {
animation: contenthighlight 1s ease;
}
@keyframes contenthighlight {
0% { border-left-color: red; }
100% { border-left-color: white; }
}
6.31.16. Go to anchor element with position:fixed header
:target:before {
content:"";
display:block;
height:90px; /* fixed header height*/
margin:-90px 0 0; /* negative fixed header height */
}
6.31.17. Center position:absolute
Don't have to know the child's exact width
.parent { position:relative; } /* center both horizontally and vertically .child { position: absolute; top: 50%; /* position the top edge of the element at the middle of the parent */ left: 50%; /* position the left edge of the element at the middle of the parent */ transform: translate(-50%, -50%); /* This is a shorthand of translateX(-50%) and translateY(-50%) */ } /* center only horizontally */ .child { position: absolute; left: 50%; transform: translateX(-50%); }
6.32. BEM
.Block__Element--Modifier- Block, Element, Modifier
.sky- parent block, standalone component
.sky__clouds- clouds element inside parent block sky
.sky .sky--dusk- a different sky. Modifier is dusk
// Block .scenery { //Elements &__sky { fill: screen; } &__ground { float: bottom; } &__people { float: center; } } //Block .sky { background: dusk; // Elements &__clouds { type: distant; } &__sun { strength: .025; } // Modifiers &--dusk { background: dusk; .sky__clouds { type: distant; } .sky__sun { strength: .025; } } &--daytime { background: daylight; .sky__clouds { type: fluffy; float: center; } .sky__sun { strength: .7; align: center; float: top; } } }
6.33. Houdini
6.33.1. Properties and Values API
- https://developer.mozilla.org/en-US/docs/Web/CSS/@property
- Support
- https://caniuse.com/mdn-css_at-rules_property
- Firefox only supports it in Nightly build
@property --colorPrimary { syntax: ''; initial-value: magenta; inherits: false; } .card { background-color: var(--colorPrimary); /* magenta */ } .highlight-card { --colorPrimary: yellow; background-color: var(--colorPrimary); /* yellow */ } .another-card { --colorPrimary: 23; background-color: var(--colorPrimary); /* @property fallback to magenta. Without @property, --colorPrimary = 23 and it's not valid for background-color and hence the browser ignores the invalid value */ }
6.34. CSS Frameworks
6.34.1. Bootstrap
6.34.1.1. Mobile Meta Setup
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <!-- Bootstrap css and js files are about 170kb --> <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous"> </head> <body> <!-- HTML Code --> <script src="https://code.jquery.com/jquery-1.12.4.min.js" integrity="sha256-ZosEbRLbNQzLpnKIkEdrPv7lOy9C27hHQ+Xp8a4MxAQ=" crossorigin="anonymous"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script> </body> </html>
6.34.1.2. Global
box-sizingis border-box including*:beforeand*:afterfont-size:16pxis declared on<html>andfont-size:1remon<body>- BS3 default font-size is 14px
- <body> also sets global font-family and line-height
- <body> has background-color #fff
- bootstrap.js requries jQuery
- By default, flexbox is not enabled
- If flexbox is enabled:
- entire grid system switches from float to flex
- input groups move from table to flex
- media components move from table to flex
- After flex is enabled, you might have problems with IE9 and IE10
6.34.1.3. Content
- Native font stack
Switched from Helvetica Neue, Helvetica and Arial to:
- -apply-system (Safari for OS X and iOS)
- BlinkMacSystemFont (Chrome for OS X)
- Segoe UI (Windows)
- Robot (Android)
- Helvetica Neue, Arial, sans-serif !default;
- h1, h2 heading elements
- heading elements
- margin-bottom: .5rem, no margin-top
- .h1 .h2
- to imitate heading display without margins
- (no term)
- Add .display-1, …, .display-4 to change the size and display for <h1>, <h2>, etc.
- (no term)
- Add <small class="text-muted">secondary heading text</small> to <h1>
- inline with text in <h1>, light color
<p>
- margin-bottom: 1rem, no margin-top
- Add .lead to <p> to make it more standout.
- Lists <ul> <ol> <dl>
- margin-bottom: 1rem, no margin-top
- Except nested lists
- add .list-unstyled to <ul>, <ol>. Only only apply for Level-1 children <li>
- add .list-inline to <ul, <ol> and .list-inline-item to li
- add .dl-horizontal to <dl>
- Horinzontal Center (text, truncate, block)
- Add .text-center in column
- Add .center-block in column (margin:0 auto)
- Add .text-truncate in column can truncate text and provides ellipsis
- Special Text
- <kbd> User input
- <kbd>cd</kbd>, <kbd><kbd>ctrl</kbd> + <kbd>,</kbd></kbd>
- (no term)
- <pre> has margin-top removed and add margin-bottom: 1rem;
- (no term)
- add .pre-scrollable to <pre> to set a max-height and make it horizontally scrollable
- <code><em></code>
- red text
- (no term)
- <samp><em></samp>
- (no term)
- add .small to imitate <small> (normally it's 85% of parent font-size)
- (no term)
- <mark> light yellow
- (no term)
- <s>, <del> strike
- (no term)
- <ins> same as <u> underline
- (no term)
- <em>, <i> and <var> look the same
- (no term)
- add .initialism to cap and make font size 90% <abbr title="full term">HTML</abbr>
- (no term)
- add .blockquote-reverse to
<blockquote><p>...</p><footer>...</footer></blockquote>
- Vertical Align
- Only applies to inline, inline-block, inline-table and table cell
- .align-baseline, .align-top, .align-middle, .align-bottom
- .align-text-bottom, .align-text-top
- Table
- Add class="table" to <table>
- add .table-inverse (dark) to inverse color
- add .table-striped to add color to even or odd row within <tbody>
- add .table-bordered
- add .table-hover
- add .table-sm (or .table-condensed) to remove cell padding
- add .table-reflow to transpose
- Add scope="row" to the first td (or th) in each <tr> inside <tbody>
- Add class="thead-inverse" or thead-default in <thead> to make table header dark or light
- Add colors to rows or individual cells. If table-inverse, use
bg-*ortext-*- .table-active (.active)
- light grey
- .table-success (.success)
- light green
- .table-info (.info)
- light blue
- .table-warning (.warning)
- light yellow
- .table-danger (.danger)
- light red
- Wrap table.table inside div.table-responsive to make table horizontal scrollable on small devices (under 768px)
- Add class="table" to <table>
- Image
.img-responsive BS3 .img-fluid BS4
<!-- block, max-width: 100%; height: auto; --> <img src="" class="img-responsive"> <!-- add .center-block to <img> to center .img-responsive --> <!-- Or center an image as inline --> <img src="" class="img-responsive img-circle" style="display:inline">
max-width and max-height
<div class="img-container" style="width:300px;height:168px;"> <img src="…"> </div> .img-container img { max-width: 100%; max-height: 100%; }
Image shapes :: img-circle, img-rounded, img-thumbnail
- Colors
text-muted, text-primary, -success, -info, -warning, -danger btn-primary, -success, -info, -warning, -danger bg-primary, -success, -info, -warning, -danger
- Quick float content .pull-left, .pull-right
Not to be used in .navbar. Use .navbar-left or .navbar-right instead.
6.34.1.4. Layout
- Grid
<div class="container"> <div class="row"> <div class="col-*-*"></div> </div> <div class="row"> <div class="col-*-*"></div> <div class="col-*-*"></div> <div class="col-*-*"></div> </div> <div class="row"> ... </div> </div>
For all sizes, 15px on each side of a column
.containerhas a fixed max-width based on different viewports..container-fluidtakes 100% of the viewport width and the same as .container.rowis a horizontal group of columns. Only columns can be immediate children of rows.15px on each side of a column
BS4 <576px >=576px >=768px >=992px >=1200px Class .col- .col-sm- .col-md- .col-lg- .col-xg- container none/auto 540px 720px 960px 1140px max width BS3 <768px >=768px >=992px >=1200px Class .col-xs- .col-sm- .col-md- .col-lg- container none/auto 750px 970px 1170px max width @media (min-width: 576px) {} => >=576@media (max-width: 576px) {} => <=576col-lg-9- non large size will have 100% width
col-md-7 col-lg-9- small size and extra small will have 100% width
col-xs-9 col-md-7- large size will have 7 columns, small size will have 9 columns
- 728px
- requires (BS4) md-12, lg-9, xg-8; (BS3) sm-12, md-9, lg-8
- 300px
- requries (BS4) md-5, lg-4, xg-3; (BS3) sm-5, md-4, lg-3
When column has 100% width, margin-bottom is 25px
- Prevent strange wrapping with uneven height content at certain viewport size
div.clearfix.visible-*-(block, inline-block)
Offset Columns
col-md-offset-*- increase the left margin of a column by n columns
Offset can be used to horizontally center a column
div.col-xs-offset-3.col-xs-6:: to the left 3 columns, width 6 columns, to the right 3 columns.col-xs-offset-3carries forward to bigger size. In order to remove offset for larger sizes, add .col-sm-offset-0Change Column Ordering
col-sm-push-*,col-sm-pull-*:: col-sm-push-8, col-sm-pull-4. Only at sm the columns are pushed/pulled. This only works when elements can be floated in one row. When they are stacked, this trick doesn't work. - Responsive Utility bs:responsive utility
.hidden-*-down- e.g. .hidden-md-down hides an element on xs, sm and md viewports
.hidden-*-up- .hidden-md-up hides an element on md, lg and xg viewports
- (no term)
- combine
.hidden-*-downand.hidden-*-upto only show element for a specific viewport size .hidden-*- * is xs, sm, lg, xl
- .hidden-print
- hide element for print device
- .visible-print-block
- only visible in print as display: block
- .visible-print-inline
- only visible in print as display: inline
- .visible-print-inline-block
- only visible in print as display: inline-block
- (no term)
.visible-*-block, -inline, -inline-block where*is xs, sm, lg, xl
- Media Object
Align media (image, video, audio) to the left or right of a content block
<div class="media"> <!-- .media-middle, .media-bottom, default is top --> <!-- .media-left, .media-right --> <a class="media-left" href="#"> <img class="media-object" src="..." alt="..."> </a> <div class="media-body"> <h4 class="media-heading">Media Heading</h4> Some Description... <!-- Nest another media object --> </div> </div>Put multiple li.media inside ul.media-list. Good for layout comments
6.34.1.5. Utilities
- Spacing Utility (Boostrap 4)
- m
- margin
- p
- padding
- t, b, l, r
- top, bottom, left, right
- x
- *-left and *-right
- y
- *-top and *-bottom
- a
- all 4 sides
- mt-0
- margin top 0
- mx-auto
- same as .center-block
In Bootstrap 3, use .form-group to create margin-bottom: 15px
- Responsive Utility
- Responsive helpers (embed and float) bootstrap:responsive helpers
Apply to <iframe>, <embed>, <video> and <object> to maintain media size ratio
<div class="embed-responsive embed-responsive-16by9"> <iframe class="embed-responsive-item" src="//www.youtube.com/embed/zpOULjyy-n8?rel=0" allowfullscreen></iframe> </div>
BS3 4by3 and 16by9. BS4 added 21by9 and 1by1
Control floats based on viewport size breakpoint float-xs-left, float-xs-right, float-xs-none
Raw CSS
<div class="videoWrapper-16-9"> <!-- Copy & Pasted from YouTube --> <iframe width="560" height="349" src="http://www.youtube.com/embed/n_dZNLr2cME?rel=0&hd=1" style="border:0" allowfullscreen></iframe> </div>
.videoWrapper-16-9 { position: relative; padding-bottom: 56.25%; /* 16:9 */ padding-top: 25px; height: 0; } .videoWrapper-16-9 iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }
- Border (BS4)
- .rounded
- .rounded-circle
- .rounded-top, right, left, bottom
- Text bs:text align
- BS4
.text-*-left,.text-*-right,.text-*-center,*is xs, sm, md, lg, xl- BS3
- .text-justify, .text-left, right, center
- Text nowrap
- .text-nowrap
- (no term)
- Text color
- .text-muted
- .text-primary, -success, -info, …
- (no term)
- Font weight
- .font-weight-bold, .font-weight-normal
- .font-italic
6.34.1.6. Components
- Alert
Wrap text in div with .alert and one of the alert color class
- alert-success
- alert-info
- alert-warning
- alert-danger
100% width block with inner and outter spacing
Non-dismissible
<div class="alert alert-danger" role="alert"> Some text with <a href="#" class="alert-link"> a link </a> some other text </div>
Dismissable
<div class="alert alert-warning alert-dismissible" role="alert"> <button type="button" class="close" data-dismiss="alert" aria-label="Close"> <span aria-hidden="true">×</span> </button> <strong>Warning!</strong> Better check yourself, you're not looking too good. </div>
- Jumbotron
.page-header :: Extra space on top and some on bottom
<div class="page-header"> <h1>Example page header <small>Subtext for header</small></h1> </div>
Jumbotron :: 100% width BS4 uses h.display-3, p.lead
<div class="jumbotron"> <h1 class="display-3">Hello, world!</h1> <p class="lead">Text.</p> <hr class="my-2"> <p>Another paragraph.</p> <p class="lead"> <a class="btn btn-primary btn-lg" href="#" role="button">Learn more</a> </p> </div> - Tag (BS4) Label (BS3)
bs:tag Used in <button>, <h1> as inline element and in bs:list group <span class="tag tag-default">New</span> <span class="label label-info">New</span>
tag-pill or badge in BS3 can collapse when there's no content <span class="tag tag-pill tag-default">Default</span>
bs:Center Tags Vertically center span inside h1 h1,h2,h3 { vertical-align:middle; }
h1>.tag, h2>.tag, h3>.tag { vertical-aign:middle; margin-top:-0.5em; font-size: 50%; * default is 75%. Make the tag smaller * }
- Progress
Contextual classes :: progress-success, etc. .progress-striped
BS4
<div class="text-xs-center" id="example-caption-4">Some Text appears on top progress bar</div> <progress class="progress" value="75" max="100" aria-describedby="example-caption-4"></progress>
BS3 is more reliable..
Contextual classes :: progress-bar-success, etc. .progress-bar-striped Add .active to .progress-bar-striped to animate the stripes right to left
<div class="progress"> <div class="progress-bar" role="progressbar" aria-valuenow="0" aria-valuemin="0" aria-valuemax="100" style="min-width: 2em;"> 0% </div> </div> <div class="progress"> <div class="progress-bar" role="progressbar" aria-valuenow="2" aria-valuemin="0" aria-valuemax="100" style="min-width: 2em; width: 2%;"> 2% </div> </div> <!-- Align multiple .progress-bar's on one line in ratio --> <div class="progress"> <div class="progress-bar progress-bar-success" style="width: 35%"> <span class="sr-only">35% Complete (success)</span> </div> <div class="progress-bar progress-bar-warning progress-bar-striped" style="width: 20%"> <span class="sr-only">20% Complete (warning)</span> </div> <div class="progress-bar progress-bar-danger" style="width: 10%"> <span class="sr-only">10% Complete (danger)</span> </div> </div>
- Button, Button Group bs:button
Used in bs:nav bs:navbar Add .btn.btn-default to <a>, <button> and <input> Only button.btn can be used as button in nav
btn-default, btn-primary, btn-success, info, warning, danger
btn-lg, btn-sm, btn-xs
btn-block :: full width of parent
Add .active to .btn to show the button is pressed
Button with icon
<a href="#" class="btn btn-default btn-lg"> <span class="glyphicon glyphicon-search"></span> Button Large </a>
bs:btn-group Use role="group" for .btn-group and .btn-group-vertical div.btn-group to align <button>s horizontally without padding nor margin div.btn-group-vertical to align <button>s vertically without padding nor margin
Use role="toolbar" for .btn-toolbar When there're multiple div.btn-group or div.btn-group-vertical, use div.btn-toolbar to wrap them.
To group a bs:dropdown with <button>s, use div.btn-group to wrap the dropdown all together.
- Form
- .sr-only
- add to <label> to hide it
- .form-group
- add to <div> which wraps each form field to add margin-bottom
- .form-group-lg, .form-group-sm
- sizing
- .form-control
- .form-control
- add to <input>, <select>, <textarea> to make it width 100%
- .form-control-file
- add to <input type=file> for 100% width
- input.form-control-lg, input.form-control-sm
- sizing (BS4)
- input.input-lg, input.input-sm
- sizing (BS3)
- Help Text (after <input>)
- .form-text (.help-block)
- block-level help text. p.form-text or div.form-text
- .text-muted
- inline help text small.text-muted or span.text-muted
- (no term)
- aria-describedby attribute (may also refer to the feedback)
<div class="form-group"> <label for="inputPassword">Password</label> <input type="password" id="inputPassword" class="form-control" aria-describedby="passwordHelpInline"> <small id="passwordHelpInline" class="text-muted"> Must be 8-20 characters long. </small> </div> - .form-inline
- By default, <form> doesn't have to have any class
- add to <form> and any form field to make it inline. Only works for md and above viewport sizes
- Checkbox and Radio
- div.checkbox (.form-check BS4), div.radio (BS3 only)
- label.checkbox-inline (.form-check-inline BS4), label.radio-inline (BS3 only)
- label.form-check-label (BS4 only)
- input.form-check-input (BS4 only)
<div class="checkbox"> <label> <input id="a" type="checkbox"> Only one checkbox </label> </div>
<div class="form-group"> <label class="checkbox-inline"> <input name="a" id="a1" type="checkbox"> Checkbox #1 </label> <label class="checkbox-inline"> <input name="a" id="a2" type="checkbox"> Checkbox #2 </label> </div>
- .input-group bs:form:input-group
Group multiple inputs of different types (or even text) in one line. Good for making very short form
<div class="form-group"> <label class="sr-only" for="exampleInputAmount">Amount (in dollars)</label> <div class="input-group"> <div class="input-group-addon"> <input type="checkbox" checked> </div> <input type="text" class="form-control" id="exampleInputAmount" placeholder="Amount"> <div class="input-group-addon">.00</div> </div> </div> <div class="form-group"> <div class="input-group"> <input type="text" class="form-control" placeholder="Search terms"> <span class="input-group-btn"> <button class="btn btn-default" type="submit">Go!</button> </span> </div> </div> - Align fields in columns
Add
.col-*-*to <label> and container of form field. Also add .control-label to <label><div class="form-group"> <label class="col-sm-2 control-label" for="inputComments">Comments</label> <div class="col-sm-10"> <textarea class="form-control" id="inputComments"></textarea> </div> </div>
<!-- Align Submit button --> <div class="form-group"> <div class="col-sm-10 col-sm-offset-2"> <input type="submit" class="btn btn-default" value="submit"> </div> </div> <!-- Align checkbox/radio --> <div class="form-group"> <div class="col-sm-10 col-sm-offset-2"> <label class="checkbox-inline"> <input name="a" type="checkbox"> Checkbox #2 </label> </div> </div>
- Validation and Feedback
Add validation class to div.form-group to add color as a whole
- .has-success
- .has-warning
- .has-error
- .has-feedback
Validation feedback with icon and/or text (only input type=text)
- .has-feedback
- add to div.form-group
- .control-label
- add to <label>
- span.glyphicon.glyphicon-ok.form-control-feedback
- icon inside field (BS3)
- input.form-control-warning, -success, -danger
- icon inside field (BS4)
- div.form-control-feedback
- feedback in text
<div class="form-group has-feedback"> <label class="control-label" for="inputName">Name</label> <input class="form-control" id="inputName" type="text" placeholder="Name"> <span class="glyphicon glyphicon-ok form-control-feedback"></span> </div>
- Thumbnail (BS3)
div.thumbnail is a block with padding 4px and border 1px. Place it inside a column for layout
- List group bs:list group
Used in bs:panel bs:list group
- Parent and child can be any HTML elements.
- <ul> <li> will not have hovering effect
- Add contextual classes to li.list-group-item
- list-group-item-success, info, warning, danger
<div class="list-group"> <!-- may use .disabled, .active --> <li class="list-group-item disabled"> <!-- may use bs:tag --> New Mails <span class="tag tag-default tag-pill float-xs-right">14</span> </li> <!-- BS4: link as a button, use <div> as parent --> <a class="list-group-item list-group-item-action"> Dapibus ac facilisis in </a> <!-- BS3: link as a button, use <div> as parent --> <button type="button" class="list-group-item"> Dapibus ac facilisis in </button> <!-- Custom Content BS3 --> <a href="#" class="list-group-item"> <h4 class="list-group-item-heading">Heading</h4> <p class="list-group-item-text">...</p> </a> <!-- Custom Content BS4 --> <a href="#" class="list-group-item list-group-item-action"> <h4 class="list-group-item-heading">Heading</h4> <p class="list-group-item-text">...</p> </a> </div> - Panel (BS3 only) bs:panel
3 rows layout 100% width with padding
Contextual classes: panel-success, …
<div class="panel panel-default"> <div class="panel-heading"> <h2 class="panel-title">Title</h2> </div> <div class="panel-body"> <p>Panel Content</p> </div> <!-- may add any HTML or BS components --> <div class="panel-footer"> <a href="#">Some Action</a> </div> </div>
.panel-collapse
- Expand and collapse a panel bs:collapse
- Place .panel-body inside .panel-collpse
- .list-group can replace .panel-body inside .panel-collapse
<!-- add role="tablist" and aria-multiselectable --> <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true"> <div class="panel panel-default"> <!-- add role="tab" and id for each panel-heading --> <div class="panel-heading" role="tab" id="headingOne"> <h4 class="panel-title"> <!-- For each .panel-title - add regular bs:collapse action button with data-parent --> <!-- This panel opens initially --> <!-- no .collapsed in <a> means it expands initially --> <!-- aria-expanded="true" not false! --> <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseOne" aria-expanded="true" aria-controls="collapseOne"> Collapsible Group Item #1 </a> </h4> </div> <!-- .collapse.in :: displays content initially --> <div id="collapseOne" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne"> <div class="panel-body"> ... </div> </div> </div> <div class="panel panel-default"> <div class="panel-heading" role="tab" id="headingTwo"> <h4 class="panel-title"> <!-- .collapsed :: collapse content initially --> <a class="collapsed" role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseTwo" aria-expanded="false" aria-controls="collapseTwo"> Collapsible Group Item #2 </a> </h4> </div> <!-- .collapse :: hides content initially --> <div id="collapseTwo" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingTwo"> <div class="panel-body"> ... </div> </div> </div> </div>
- Card, Card Group, Card Deck (BS4) bs:card
Panel is replaced by Card in BS4
<!-- add .card-inverse to make text color white. contextual classes :: card-primary, info, etc. outline color :: card-outline-primary, info, etc. --> <div class="card"> <!-- .card-header in <h*> --> <h3 class="card-header">Featured</h3> <!-- header can have bs:nav, tabs or pills --> <!-- just add .card-header-tabs ---> <div class="card-header"> <ul class="nav nav-tabs card-header-tabs"> ... </ul> </div> <!-- .card-img without -top or -bottom centers the image inside .card --> <img class="card-img-top" src="..." alt="Card image cap"> <!-- Overlay content on image --> <div class="card-img-overlay"> <h4 class="card-title">Image title</h4> <p class="card-text">...</p> <p class="card-text"><small class="text-muted">...</small></p> </div> <!-- Wrap text with .card-block --> <div class="card-block"> <h4 class="card-title">Card title</h4> <p class="card-text">...</p> <!-- regular button --> <a href="#" class="btn btn-primary">Go</a> </div> <div class="card-block"> <!-- .card-title and .card-subtitle in <h*> --> <h4 class="card-title">Card title</h4> <h6 class="card-subtitle text-muted">Support card subtitle</h6> </div> <!-- bs:list group --> <ul class="list-group list-group-flush"> <li class="list-group-item">Cras justo odio</li> <li class="list-group-item">Dapibus ac facilisis in</li> <li class="list-group-item">Vestibulum at eros</li> </ul> <div class="card-block"> <!-- .card-link --> <a href="#" class="card-link">Card link</a> <a href="#" class="card-link">Another link</a> </div> <img class="card-img-bottom" src="..." alt="Card image cap"> <div class="card-footer">2 days ago</div> <!-- .card-header in <h*> --> <h3 class="card-footer">2 days ago</h3> </div>
Card Group
- wrap all .card inside div.card-group
- each .card will have equal height with no spacing like in a table
Card Deck
- Like .card-group but with spacing
- wrap all .card inside div.card-deck
- if flex is not enabled, wrap div.card-deck with div.card-deck-wrapper
Card Columns
- Masonry
- Wrap all .card inside div.card-columns
- Dropdown (js)
bs:dropdown For navigation purpose. Used in bs:nav bs:navbar bs:btn-group
<!-- .open :: dropdown state is always open --> <div class="dropdown open"> <!-- .dropdown-toggle turn something into a dropdown toggle --> <!-- data-toggle :: attach js dropdown event --> <button type="button" id="dropdownMenuButton" class="btn btn-secondary dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> Dropdown button <span class="caret"></span> <!-- BS4 does't have to define caret --> </button> <!-- BS4 no longer requires menu items to be <li><a /></li> but you have to use .dropdown-item --> <!-- <ul> <li> <a> are used in BS3 --> <div class="dropdown-menu" aria-labelledby="dropdownMenuButton"> <h6 class="dropdown-header">Combo A (not selectable)</h6> <a class="dropdown-item" href="#">1</a> <a class="dropdown-item" href="#">2</a> <a class="dropdown-item" href="#">3</a> <!-- BS3 use .divider --> <div role="separator" class="dropdown-divider"></div> <h6 class="dropdown-header">Combo B (not selectable)</h6> <a class="dropdown-item" href="#">4</a> <!-- disabled --> <a class="dropdown-item disabled" href="#">5 (Sold out!)</a> <a class="dropdown-item" href="#">6</a> </div> </div>
To group a bs:dropdown with <button>s, use div.btn-group to wrap the dropdown all together.
<div class="btn-group" role="group" aria-label="..."> <button type="button" class="btn btn-info">Button #1</button> <button type="button" class="btn btn-info">Button #2</button> <!-- dropdown starts here --> <div class="btn-group" role="group"> <button type="button" class="btn btn-info dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> Button #3: Dropdown <!-- BS4 doesn't need to manually add a caret --> <!-- <span class="caret"></span> --> </button> <!-- add ul.dropdown-menu mentioned above --> </div> </div>
- Navigation (js), tab (js)
- .nav, .nav-tabs, .nav-pills, tablist (js)
bs:nav :: May use
- bs:dropdown as a nav item
- bs:tab to show and hide content
- <a href="#home">
Items are blocks float to left or stacked vertically
<!-- nav-tabs or nav-pills --> <!-- .nav-stacked :: add to ul --> <!-- .nav-justified :: add to ul. Center tabs/pills --> <ul class="nav nav-tabs"> <!-- BS4 for <li> :: add .nav-item and remove role="presentation" --> <li role="presentation" class="active"> <!-- BS4 for <a> :: add .nav-link --> <a href="#">Item 1</a> </li> <li role="presentation" class="disabled"><a href="#">Item 2</a></li> <li role="presentation"><a href="#">Item 3</a></li> <!-- Item can be a bs:dropdown --> <li role="presentation" class="dropdown"> <button class="dropdown-toggle" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> Item 4 </button> <ul class="dropdown-menu"> ... </ul> </li> </ul>bs:tab Display content when nav-tabs or nav-pills is clicked $().tab data-toggle="tab"
<!-- add role="tablist" --> <ul class="nav nav-tabs" role="tablist" id="myTab"> <!-- Be sure to add .active --> <!-- BS4 for <li> :: add .nav-item and remove role="presentation" --> <li role="presentation" class="active"> <!-- <a> :: add role="tab", data-toggle="tab", aria-controls="..." BS4 :: add .nav-link may use data-target instead of href --> <a href="#item1" data-toggle="tab" role="tab" aria-controls="item1">Item 1</a> </li> <li role="presentation" class="active"> <!-- <a> :: add role="tab", data-toggle="tab", aria-controls="..." BS4 :: add .nav-link --> <a href="#item1" data-toggle="tab" role="tab" aria-controls="item2">Item 2</a> </li> </ul> <div class="tab-content"> <!-- effect :: .fade to div.tab-pane --> <div class="tab-pane active" id="item1" role="tabpanel"> Item 1 content </div> <div class="tab-pane active" id="item2" role="tabpanel"> Item 2 content </div> </div> - Navbar (js)
bs:navbar is responsive Use bs:collapse
div.collapse collpses at viewport <768px which is xs in BS3 and sm in BS4
Background image to navbar
@media (min-width:768px) { .navbar { background:url('../images/nav-bg.jpg') center repeat-x #222; background-size:auto 100%; } } @media (max-width:768px) { .navbar .navbar-header{ background:url('../images/nav-bg.jpg') center repeat-x #222; background-size:auto 100%; } }<!-- .navbar-fixed-bottom | .navbar-fixed-top :: add to <nav> if navbar is fixed --> <!-- .navbar-default | .navbar-inverse (black and white) --> <!-- customize color, border using .navbar-default --> <nav class="navbar navbar-default"> <!-- Without div.container, the navbar will align to left --> <div class="container"> <!-- Add branding --> <div class="navbar-header"> <!-- Style toggle button --> <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#myNavbar" aria-expanded="false"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="">Website Name</a> <!-- Add a logo in a.navbar-brand. Height should 20px. --> <p class="navbar-text">optional: extra text</p> </div> <!-- .collapse is display: none; --> <div class="collapse navbar-collapse" id="myNavbar"> <ul class="nav navbar-nav"> <li><a href="#">Submenu 1</a></li> <li><a href="#">Submenu 2</a></li> <!-- bs:dropdown --> <li> <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Submenu 3</a> <ul class="dropdown-menu"> ... </ul> </li> </ul> <!-- ul.navbar-nav ends --> <!-- Add extra text --> <p class="navbar-text">optional: extra text</p> <!-- Add a bs:button --> <button type="button" class="btn btn-default navbar-btn">sign in</button> <!-- Add a bs:form or bs:form:input-group --> <form class="navbar-form navbar-right" role="search"> <div class="form-group"> ... </div> </form> </div> <!-- div.collapse ends --> </div> <!-- div.container ends --> </nav> - Breadcrumb
Simple align links horizontally
<ol class="breadcrumb"> <li class="breadcrumb-item"><a href="#">Home</a></li> <li class="breadcrumb-item"><a href="#">Library</a></li> <li class="breadcrumb-item active">Data</li> </ol>
- Pagination and pager
Simply align links horizontally BS3 recommends to replace <a> with <span> when li.disabled
<nav aria-label="Page navigation"> <!-- .pagination-lg | .pagination-sm :: add to ul --> <ul class="pagination"> <!-- li.disabled --> <li class="disabled"> <a href="#" aria-label="Previous"> <span aria-hidden="true">«</span> </a> </li> <!-- li.active --> <li class="active"> <a href="#">1</a> <span class="sr-only">(current)</span> </li> <li><a href="#">2</a></li> <li><a href="#">3</a></li> <li><a href="#">4</a></li> <li><a href="#">5</a></li> <li> <a href="#" aria-label="Next"> <span aria-hidden="true">»</span> </a> </li> </ul> </nav>
Pager :: Simple Previous and Next buttons with no pages
<!-- add <nav>, classes li.previous, li.next to align buttons to both ends --> <!-- remove <nav>, classes li.previous, li.next to center them --> <nav aria-label="..."> <ul class="pager"> <li class="previous disabled"> <a href="#"><span aria-hidden="true">←</span> Older</a> </li> <li class="next"> <a href="#">Newer <span aria-hidden="true">→</span></a> </li> </ul> </nav>
BS4 structure is a little different
- .page-item
- add to every <li>
- .page-link
- add to every li>a
- .disabled
- add to li and add tabindex="-1" to li>a. <a> will have pointer-events: none.
- Optionally remove <a> if li.disabled
- BS3 seems to recommend this behavior
- .nav, .nav-tabs, .nav-pills, tablist (js)
- Carousel (js)
Don't add data-ride if it's triggered by javascript
Pictures have to be the same width and height.
<!-- Options data-interval :: 5000 data-pause :: "hover" | null data-wrap :: true keyboard :: true --> <div id="carousel-example-generic" class="carousel slide" data-ride="carousel"> <ol class="carousel-indicators"> <!-- An item must have .active --> <li data-target="#carousel-example-generic" data-slide-to="0" class="active"></li> <li data-target="#carousel-example-generic" data-slide-to="1"></li> <li data-target="#carousel-example-generic" data-slide-to="2"></li> </ol> <div class="carousel-inner" role="listbox"> <!-- An item must have .active --> <!-- BS4 :: use .carousel-item instead of .item --> <div class="item active"> <img src="..." alt="..."> <div class="carousel-caption"> <h3>...</h3> <p>...</p> </div> </div> <div class="item"> <img src="..." alt="..."> <div class="carousel-caption"> ... </div> </div> <!-- more items --> </div> <!-- Controls --> <!-- data-slide could be prev, next, cycle, pause, number (integer starting from 0) --> <a class="left carousel-control" href="#carousel-example-generic" role="button" data-slide="prev"> <span class="glyphicon glyphicon-chevron-left" aria-hidden="true"></span> <span class="sr-only">Previous</span> </a> <a class="right carousel-control" href="#carousel-example-generic" role="button" data-slide="next"> <span class="glyphicon glyphicon-chevron-right" aria-hidden="true"></span> <span class="sr-only">Next</span> </a> </div> <script> $(function() { $('#carousel-example-generic').carousel({ // options :: see above }); /* Other methods refer to data-slide e.g. .carousel('cycle') ... */ // Catch events // slide.bs.carousel :: start // slid.bs.carousel :: complete // Event object // direction :: 'left' | 'right' // relatedTarget :: DOM element that is about to be active $('#carousel-example-generic').on('slide.bs.carousel', function(e) { // do something }); }); </script> - Modal (js)
<!-- When a modal is open, it adds .modal-open to <body> and also a .modalbackdrop to provide a click area to dismiss the modal ---> <button type="button" class="btn btn-primary btn-lg" data-toggle="modal" data-target="#myModal"> Launch demo modal </button> <!-- remove .fade to remove animation --> <div class="modal fade" id="myModal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel"> <!-- Sizing :: in modal-dialog, add modal-lg, modal-sm --> <div class="modal-dialog" role="document"> <div class="modal-content"> <div class="modal-header"> <button type="button" class="close" data-dismiss="modal" aria-label="Close"> <span aria-hidden="true">×</span> </button> <h4 class="modal-title" id="myModalLabel">Modal title</h4> </div> <div class="modal-body"> <!-- stack .row to use grid --> <!-- Any HTML or BS components --> <!-- form field autofocus has no effect. Refer to javascript below --> </div> <div class="modal-footer"> <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> <button type="button" class="btn btn-primary">Save changes</button> </div> </div> </div> </div> <script> /* Options :: data-* backdrop :: true. Use 'static' to not close the modal on click keyboard :: true. Esp key closes the modal show :: true. Show when initialized remote :: deprecated. */ /* Methods .modal(options) method_name :: toggle, show, hide, and handUpdate (readjust positioning. Only needed if modal height changes while it opens) .modal('method_name') */ /* Event types: show.bs.modal :: start. shown.bs.modal :: finished (after transition is complete) hide.bs.modal :: start to hide hidden.bs.modal :: finished being hidden loaded.bs.modal :: content has been loaded through remote option Event object e.relatedTarget :: If triggered by a click, it's the clicked element */ $('#myModal').on('show.bs.modal', function(e) { // Using the same modal to display different content var button = $(e.relatedTarget); var data = button.data('lili-content'); // Do some Ajax var modal = $(this); modal.find('.modal-title').text('...'); modal.find('.modal-body').innerHTML = '...'; <!-- form field autofocus has no effect. Use this workaround --> $('#fieldInputID').focus(); }); </script>
- Collapse (js)
bs:collapse :: Button to expand or collapse content Used in bs:navbar bs:panel bs:card (accordion)
<!-- Action Button as <a> --> <a class="btn btn-primary" role="button" data-toggle="collapse" href="#collapseExample" aria-expanded="false" aria-controls="collapseExample"> Link with href </a> <!-- Action Button as <button> --> <button class="btn btn-primary" type="button" data-toggle="collapse" data-target="#collapseExample" aria-expanded="false" aria-controls="collapseExample"> Button with data-target </button> <!-- .collapse :: hide content initially --> <div class="collapse" id="collapseExample"> <!-- BS4 :: use .card.card-block instead of .well --> <div class="well"> ... </div> </div> - Tooltip (js)
Need to initiate in javascript
<button type="button" class="btn btn-default" data-toggle="tooltip" data-placement="left" title="Tooltip on left">Tooltip on left</button> <a href="#" data-toggle="tooltip" data-placement="bottom" title="">Tooltip on bottom</a> <script> $('[data-toggle="tooltip"]').tooltip(); </script> - Popover (js)
Need to initiate in javascript A title can be specified
data-trigger can be click | hover | focus | manual
<a tabindex="0" class="btn btn-lg btn-danger" role="button" data-toggle="popover" data-container="body" data-trigger="focus" title="Dismissible popover" data-content="And here's some amazing content. It's very engaging. Right?"> Click on anywhere to dismiss the popover </a> <button type="button" class="btn btn-lg btn-danger" role="button" data-toggle="popover" data-trigger="focus" title="Dismissible popover" data-content="And here's some amazing content. It's very engaging. Right?"> Must click on the button again to dismiss the popover </button> <script> $('[data-toggle="popover"]').popover(); </script> - Scrollspy (js)
Must use with bs:nav
<body data-spy="scroll" data-target="#navbar-example"> ... <div id="navbar-example"> <ul class="nav nav-tabs" role="tablist"> ... </ul> </div> ... </body> <style> body { position: relative; } </style>
6.34.1.7. Mockup Tools
6.34.1.8. v4
- Display utilities
.d-*.display-*
.d-{value} for xs .d-{breakpoint}-{value} for sm, md, lg, and xl.
d-none d-inline d-inline-block d-block d-table d-table-cell d-table-row d-flex d-inline-flex
Hidden on all .d-none Hidden only on xs .d-none .d-sm-block Hidden only on sm .d-sm-none .d-md-block Hidden only on md .d-md-none .d-lg-block Hidden only on lg .d-lg-none .d-xl-block Hidden only on xl .d-xl-none Visible on all .d-block Visible only on xs .d-block .d-sm-none Visible only on sm .d-none .d-sm-block .d-md-none Visible only on md .d-none .d-md-block .d-lg-none Visible only on lg .d-none .d-lg-block .d-xl-none Visible only on xl .d-none .d-xl-block
Display headings <h1 class="display-1">Display 1</h1> <h1 class="display-2">Display 2</h1> <h1 class="display-3">Display 3</h1> <h1 class="display-4">Display 4</h1>
- p.lead
.lead { font-size: 1.25rem; font-weight: 300; } <p class="lead">25% bigger than 1rem and 300 font weight.</p> Can be used to enlarge a button <p class="lead"> <a class="btn btn-primary btn-lg" href="#" role="button">Learn more</a> </p> - Spacing
mt, pt, mb, pb mx, px for left and right my, py for top and bottom
{property}{sides}-{size} for xs {property}{sides}-{breakpoint}-{size} for sm, md, lg and xl
size m-0 m-1 to $spacer * .25 m-2 $spacer * .5 m-3 $spacer m-4 $spacer * 1.5 m-5 $spacer * 3 m-auto
$spacer is 1 rem
- flex
- Use case
Left and right
<section class="search-bar"> <div class="page-wrap container"> <div class="d-flex flex-wrap align-items-center"> <h1 class="mr-auto p-2 text-white text-uppercase">New And Used Inventory</h1> <a href="/admin" class="p-2 btn btn-light btn-sm bg-white text-primary" role="button">Adminstrator Login</a> </div> </div> </section> - display
.d-flex .d-inline-flex .d-sm-flex .d-sm-inline-flex .d-md-flex .d-md-inline-flex .d-lg-flex .d-lg-inline-flex .d-xl-flex .d-xl-inline-flex
- flex-row
.flex-row .flex-row-reverse .flex-column .flex-column-reverse .flex-sm-row .flex-sm-row-reverse .flex-sm-column .flex-sm-column-reverse .flex-md-row .flex-md-row-reverse .flex-md-column .flex-md-column-reverse .flex-lg-row .flex-lg-row-reverse .flex-lg-column .flex-lg-column-reverse .flex-xl-row .flex-xl-row-reverse .flex-xl-column .flex-xl-column-reverse
- flex-wrap
.flex-nowrap .flex-wrap .flex-wrap-reverse .flex-sm-nowrap .flex-sm-wrap .flex-sm-wrap-reverse .flex-md-nowrap .flex-md-wrap .flex-md-wrap-reverse .flex-lg-nowrap .flex-lg-wrap .flex-lg-wrap-reverse .flex-xl-nowrap .flex-xl-wrap .flex-xl-wrap-reverse
- justify-content
.justify-content-start .justify-content-end .justify-content-center .justify-content-between .justify-content-around .justify-content-sm-start .justify-content-sm-end .justify-content-sm-center .justify-content-sm-between .justify-content-sm-around .justify-content-md-start .justify-content-md-end .justify-content-md-center .justify-content-md-between .justify-content-md-around .justify-content-lg-start .justify-content-lg-end .justify-content-lg-center .justify-content-lg-between .justify-content-lg-around .justify-content-xl-start .justify-content-xl-end .justify-content-xl-center .justify-content-xl-between .justify-content-xl-around
- align-items
.align-items-start .align-items-end .align-items-center .align-items-baseline .align-items-stretch .align-items-sm-start .align-items-sm-end .align-items-sm-center .align-items-sm-baseline .align-items-sm-stretch .align-items-md-start .align-items-md-end .align-items-md-center .align-items-md-baseline .align-items-md-stretch .align-items-lg-start .align-items-lg-end .align-items-lg-center .align-items-lg-baseline .align-items-lg-stretch .align-items-xl-start .align-items-xl-end .align-items-xl-center .align-items-xl-baseline .align-items-xl-stretch
- auto margin
<!-- all three left --> <div class="d-flex"> <div class="p-2">Flex item</div> <div class="p-2">Flex item</div> <div class="p-2">Flex item</div> </div> <!-- left right right--> <div class="d-flex"> <div class="mr-auto p-2">Flex item</div> <div class="p-2">Flex item</div> <div class="p-2">Flex item</div> </div> <!-- left left right --> <div class="d-flex"> <div class="p-2">Flex item</div> <div class="p-2">Flex item</div> <div class="ml-auto p-2">Flex item</div> </div>
- iOS 8 and lower
v4 doesn't support flex for iOS 8 and lower.
display: -webkit-box; /* v4 uses. and this doesn't support wrap so -webkit-flex has to be used */ display: -moz-box; display: -ms-flexbox; display: -webkit-flex; /* should use this */ display: flex; /* v4 doesn't use any -webkit-flex-* */ -webkit-flex-wrap: wrap; -webkit-justify-content: center; justify-content: center; -webkit-align-items: center; align-items: center;Refer to modernizr:flexbox
- Use case
- color
https://getbootstrap.com/docs/4.0/utilities/colors/
.bg-white .text-white
primary :: blue secondary :: grey success :: green danger :: red warning :: yellow info :: cyan light :: light grey muted :: darker grey similary to secondary
- text
.text-lowercase -uppercase -capitalize
- Popover
- Require popper.js
npm install --save popper.js
Need to be initialized using javascript
// initialize all $(function () { $('[data-toggle="popover"]').popover() }) //
button not dismissable
<button type="button" class="btn btn-lg btn-danger" data-toggle="popover" title="Popover title" data-content="And here's some amazing content. It's very engaging. Right?">Click to toggle popover</button>
<a> dismissable (click somewhere else)
<a tabindex="0" class="btn btn-lg btn-danger" role="button" data-toggle="popover" data-trigger="focus" title="Dismissible popover" data-content="And here's some amazing content. It's very engaging. Right?">Dismissible popover</a>
Events :: *.bs.popover show, shown, hide, hidden, inserted
$('#myPopover').on('shown.bs.popover', function () { // do something… })
Methods
trigger mode manual if you want to distinguish hover and click events
$('.action-call[data-toggle="popover"]').popover({trigger:'manual'}) .on({ mouseenter: function() {$(this).popover('show');}, mouseleave: function() {$(this).popover('hide');}, click: function() {window.open('tel:'+$(this).data('phone'));} });
6.34.1.9. Third party
6.34.2. Tailwind
6.34.2.1. Install
- Via Tailwind CLI
npm i -D tailwindcss npx tailwindcss init # Generated src/css/tailwind.config.js
- Via PostCSS
npm install -D tailwindcss postcss autoprefixer npx tailwindcss init # Edit postcss.config.js
import tailwind from 'tailwindcss' import autoprefixer from 'autoprefixer' import tailwindConfig from './src/css/tailwind.config.js' export default { plugins: [tailwind(tailwindConfig), autoprefixer], }
- Playground
6.34.2.2. Config
- Default config and default theme
- Color
node_modules/tailwindcss/lib/public/colors.js- https://tailwindcss.com/docs/customizing-colors
6.34.2.3. padding
- p-1
- 0.25rem
=4px - p-2
- 0.5rem
=8px - p-4
- 1rem
6.34.2.4. About turning off preflight
6.34.2.5. Pseudo-class Reference
- All
- https://tailwindcss.com/docs/hover-focus-and-other-states#pseudo-class-reference
- (no term)
hover:- (no term)
focus:focus_within:focus:visible
- (no term)
active:- (no term)
visited:- (no term)
target:- URL #name open
- (no term)
selection:- (no term)
<li class="marker:text-pink-500" list-disc>marker:- the list marker (bullet/number)
list-disc- disk (dot) style
list-decimal- number
list-none- no bullet
6.34.2.6. Position modifiers
first:first-of-type:
last:last-of-type:
only:only-of-type:
odd:even:empty:empty:hidden- Hide if there's no content
6.34.2.7. Form modifiers
- HTML state and corresponding Tailwind modifier
- disabled
disabled:- (no term)
checked:indeterminate:- group of radio buttons or checkboxes
- (no term)
valid:invalid
- (no term)
in-range:out-of-range:
file:hover:file:
6.34.2.8. Child Modifiers and Sybling Modifiers
group:group-hover:
peer:peer-*:peer-invalid:
6.34.2.9. Attribute modifier
<html dir="ltr">ltr:rtl:
<details class="group open:ring-2 open:p-3">open:group-open:
6.34.2.10. Media modifier
portait:landscape:print:
6.34.2.11. Dark mode
dark:
6.34.2.12. Arbitrary Values
<div class="xl:w-[80%]"></div> <!-- Tailwind in some cases can auto resolve ambiguities --> <!-- Will generate a font-size utility --> <div class="text-[22px]">...</div> <!-- Will generate a color utility --> <div class="text-[#bada55]">...</div> <!-- To resolve ambiguities manually, specify the CSS key or CSS data type --> <p class="text-[color:var(--primary)] text-[clamp(1em,4vw,4em)]">Hello</p>
6.34.2.13. Tools built on top of Tailwind
- Tailwind UI
- components and templates
- Some are not free
- Tailwind Kit
- https://www.tailwind-kit.com/
- components and templates
- Free
- Headless UI
- components and templates
- Free
- Daisy UI
- https://daisyui.com/
- CSS classes
6.34.3. MUI
- https://www.figma.com/community/file/912837788133317724
- Base UI e.g.
@mui/base/Buttonis a library of unstyled React UI components and hooks - MUI System is a set of CSS utilities that provide a styled-components-like API
- Material UI e.g.
@mui/material/buttonis Base UI plus styles - MUI X is a collection of advanced UI components and should install with Material UI
6.34.3.1. MUI System
@mui/system
- styled
- Add prop
sxto non-MUI-component
import { styled } from '@mui/material/styles'; const Div = styled('div')``;
- Add prop
- unstable_styleFunctionSx
import * as React from 'react'; import styled, { ThemeProvider } from 'styled-components'; import { unstable_styleFunctionSx } from '@mui/system'; import { createTheme } from '@mui/material/styles'; const theme = createTheme(); const Div = styled('div')(unstable_styleFunctionSx); export default function StyleFunctionSxDemo() { return ( <ThemeProvider theme={theme}> <Div sx={{ m: 1, p: 1, border: 1 }}>Custom component with the sx prop</Div> </ThemeProvider> ); }
6.34.4. Pico.css
- https://picocss.com/
- Classless version or less amount of classes version
- One CSS file, no JavaScript
- Basic components
- Modal
- Accordion
- Card
- Dropdown
- Nav
- Progress
- Loading
- Tooltip
6.34.5. Animate.css
- https://animate.style/
- Can preview each animation effect
- https://github.com/animate-css/animate.css
7. HTML
7.1. DOM
7.1.1. Window
- pageYOffset
- scrollY
- same as dom:window:pageyoffset
- innerHeight
window.innerHeight + window.pageYOffsetis the y coordinate of the window
7.1.2. Document Object
Document properties and methods
| document.methodName() or | Notes | Use in |
|---|---|---|
| document.propertyName | html:ElementObject | |
| createElement("p") | appendChild | no |
| createTextNode("text node"); | appendChild | no |
| createAttribute('data-id') | appendChild | no |
| getElementById("id") | null or html:ElementObject | no |
| getElementsByClassName | NodeList | yes |
| getElementsByTagName("p") | NodeList | yes |
| querySelector('#id') | 1st matched element. null or exception or html:ElementObject | yes |
| querySelectorAll('#id') | Static NodeList or exception | yes |
| normalize() | remove empty text nodes and join adjacent text nodes | yes |
| readyState | - loading :: the document is still loading | no |
| html:document:readyState | - interactive :: document has finished loading and | |
| the document has been parsed but sub-resources e.g. | ||
| images, stylesheets and frames are still loading | ||
| - complete :: document and all sub-resources have finished | ||
| loading. The state indicates that the html:event:load | ||
| is about to fire |
- designMode
- default
"off"
7.1.3. NodeList Object NodeList
NodeList Object appears like an array (loop like an array) but cannot use Array Methods such as valueOf(), join(), forEach().
Static NodeList means changes to DOM have no effect.
All NodeList objects are live objects meaning changes that are made in other code places reflect on all NodeList objects that were previously found.
var myNodeList = document.getElementsByTagName("P"); console.log(myNodeList.length); var firstP = myNodeList[0]; var firstP = myNodeList.item(0); / same as above, return null if not exists var firstP = document.getElementsByTagName("p")[0]; / same as above
console.log(firstP.innerHTML); firstP.style.backgroundColor = "red";
for (var i=0; i < myNodeList.length; i++) { myNodeList[i].style.backgroundColor = "red"; }
Other ways to loop NodeList
Array.prototype.forEach.call( aNodeListArray, function(item) { console.log(item.id); } ); [...myNodeList].forEach( function(item) { console.log(item.id); } );
7.1.4. Element Object
7.1.4.1. Basics
Element Properties and Methods
| element.methodName() or | Notes | Use in |
|---|---|---|
| element.propertyName | html:DocumentObject? | |
| accessKey | get/set accessKey | |
| addEventListerner() | refer to event attributes without 'on' | yes |
| e.addEventListerner("click", myFunction,false ) | ||
| false for bubbling, true for capturing | ||
| removeEventListener() | anonymous function will not work | yes |
| click() | simulate a click on an element | no |
| blur() | no | |
| focus() | no | |
| innerHTML | get/set | no |
| appendChild(newChildNode) | move element to the end of another element | no |
| appendChild | ||
| removeChild(childNode) | no | |
| list.insertBefore(newnode, | ||
| list.childNodes[0]) | ||
| parentNode | read only | no |
| parentElement | read only | no |
| .cloneNode() | ||
| attributes | read only. NamedNodeMap | no |
| getAttribute("class") | attribute value in string | no |
| getAttributeNode("class") | Attr object | no |
| .childElementCount | the number of child element nodes not text | |
| and comment nodes | ||
| .childNodes | NodeList object. .childNodes.length | |
| var c = e.childNodes[0].text | ||
| .children | exclude text and comment nodes. | |
| HTMLCollection object. | ||
| classList | read only. DOMTokenList | no |
| className | get/set | no |
| clientHeight | height + padding, no border and scrollbar | no |
| clientWidth | no | |
| clientTop | Top border width | no |
| clientLeft | Left border width | no |
| offsetWidth | width+padding+border+scrollbar but no margin | no |
| offsetHeight | no | |
| .offsetLeft, offsetTop | relative to offsetParent | |
| .scrollHeight, .scrollWidth | width+padding only | |
| .scrollLeft, .scrollTop | pixels are scrolled | |
| e1.compareDocumentPosition(e2) | compare positions of elements | |
| e1.contains(e2) | if e2 is a descendant of e1 | |
| .contentEditable | ||
| .firstChild | child node | |
| .firstElementChild | child element node | |
| .nextSibling | ||
| .nextElementSibling | ||
| .hasAttribute("onclick") | ||
| .hasAttributes() | has any attributes? | |
| .hasChildNodes() | has any child nodes | |
| .isEqualNode(e1,e2) | ||
| .nodeName | tag name | |
| .tagName | tag name in upppercase | |
| .nodeValue | ||
7.1.4.2. Reflow
- Reflow is a user-blocking operation that computes the layout of the document. A reflow on an element is the calculation of its dimensions and position in the document
- Insert, remove, update an element in the DOM
- modify content in an element including text in input box
- Move and animate a DOM element
- take measurements of an element e.g. offsetWidth and getComputedStyle
- Change a CSS style
- Set
styleattribute - change font
- Set
- Activate a CSS-pseudo class e.g.
:hover - add or remove a style sheet
- resize the window
- scroll
- https://csstriggers.com/ lists which CSS property change triggers what types of reflow
- layout, paint or composite
- https://gist.github.com/paulirish/5d52fb081b3570c81e3a lists JavaScript codes that trigger a reflow
- https://medium.com/better-programming/web-performance-dom-reflow-76ac7c4d2d4f
- Insert, remove, update an element in the DOM
7.1.4.3. elem.offsetWidth, elem.clientWidth, elem.scrollWidth, getComputedStyle(elem).width, elem.getBoundingClientRect().width
offsetWidth,offsetHeight- width + padding + border + scrollbar, but no margin
- Rounded to integer in pixels
- it is faster than
clientWidth
clientWidth,clientHeight- height + padding, but no margin, no border, no scrollbar
- 0 for inline elements and elements with no CSS
- rounded to integer in pixels
- scrollWidth
- width + padding, but no margin, no border, no scrollbar
- similar to clientWidth
- (no term)
- rounded to integer in pixels
getComputedStyle(elem).width- https://developer.mozilla.org/en-US/docs/Web/API/Window/getComputedStyle
returns a live CSSStyleDeclaration object
var style = window.getComputedStyle(element [, pseudoElt]);- updates automatically when the element's styles are changed
.widthis a string withpxat the end
- could be
autofor an inline element
elem.getBoundingClientRect().width- https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect
Returns a DOMRect object which has 4 keys that hold fractional values
- size of an element (
width,height) and its position relative to the viewportxeq.left,yeq.top,right,bottom
- Returned size
widthis very close tooffsetWidthbut different- width or height + padding + border-width,
- width or height only if
box-sizing: border-box - If the element is transformed e.g.
width: 100px; transform: scale(0.5);, it will return 50 as the width
- size of an element (
- Also works for virtual DOM e.g. Vue
this.$refs['some-ref'].getBoundingClientRect().height
7.1.5. Event Object
7.1.5.1. Basics
<input id="elem" type="button" value="Click me"> // event is the first argument <script> var elem = document.getElementById("elem"); elem.onclick = function(e) { alert('Thank you'); }; elem.setAttribute('onclick', "javascript:console.log('hello')"); elem.addEventListener('click', function (e) { // e is SomeEvent object }); </script> // Pass event for inlince onclick <pre onclick="doSth(event)"> function doSth(e) { e.stopPropagation(); } <pre onclick="event.stopPropagation();doSth();">
Property
| type | event name. e.g. mousedown |
| timeStamp | event occured |
| bubbles | true or false |
| defaultPrevented | if method preventDefault() was called |
| currentTarget | the element event happens on |
| target | the (child) element event happens on |
| view | get reference of the Window object where the event occured |
Method
| preventDefault() | stop defeault event action |
| stopImmediatePropagation() | stop other listeners of the same event (e.g. click) on the same target |
| stopPropagation() | stop bubbling |
7.1.5.2. Dispatch event, trigger event
<input id="lili" type="text" onchange="console.log('value is changed. New value' + this.value, event);"> <button onclick="document.getElementById('lili').value=123;document.getElementById('lili').onchange();">Change me, event object is not passed onto #lili.onchange</button> <button onclick="document.getElementById('lili').value=321;document.getElementById('lili').dispatchEvent(new Event('change'));">Change me, event object is passed onto #lili.onchange, just like a normal UI operation</button>
7.1.5.3. onclick
- Consider assign the same event handler for keydown event for devices such as TV's which don't use mouse nor a touch screen
- Returning false means the click does not go through
7.1.5.4. onchange
- onchange actually happens after oninput and then after onblur
- iOS doesn't trigger onchange if the input value is changed programmatically
In Safari, if onblur modifies the input value programmatically, and the value is different from the last oninput, then onchange will not fire!
// onblur event handler // may change text.value based on some conditions var isSafari = /^((?!chrome|android).)*safari/i.test(navigator.userAgent); if (focus && text.value !== oldAmount && text.onchange && isSafari) { text.dispatchEvent(new Event('change')); }
7.1.5.5. Events about checkbox
<div class="inputrow"> <input type="checkbox" onchange="" onclick="console.log(this.checked);" id="aCheckbox" value="1" checked> <label for="aCheckbox">Checkbox Label</label> </div>
- Initially, the checkbox is checked
- If the label or the checkbox is clicked, onclick is triggered and logs
this.checkedis false- return false in onclick will stop saving this.checked = false,
this.checkedwill remain as true- and onchange is not run
- return false in onclick will stop saving this.checked = false,
7.1.5.6. Events about radio
<input type="radio" name="my_radio" id="my_radio_0" onclick="doSomething()" value="0"> <label for="my_radio_0">None</label>
7.1.5.7. MouseEvent Object
| altKey | whether ALT key was pressed |
| shiftKey | whether Shift key was pressed |
| ctrlKey | whether Ctrl key was pressed |
| button | 0 left click, 1 middle, 2 right |
| which | button + 1: 0: no button, 1 left, 2 middle, 3 right |
| buttons | one or more buttons. More button such as browse back button.Safari not supported |
| clientX | relative to current window (viewport). |
| clientY | |
| pageX | relative to the fully rendered content area. Height might be larger than |
| pageY | viewport height. |
| screenX | relative to the screen. e.g. number of monitors and monitor screen |
| screenY | |
| relatedTarget | use with mouseover event to find the element the cursor jsut exited or |
| with the mouseout event to find the element the cursor just entered |
- onclick on
<a href="#">a tagonclick="doSomething();return false"- eq. to applying both
event.preventDefault(); event.stopPropagation(); - (no term)
onclick="doSomething(event);"function doSomething(e, myVal){ //doing custom things with myVal //here I want to prevent default e = e || window.event; e.preventDefault(); // e.stopPropagation(); }
7.1.5.8. KeyboardEvent Object
- Keyboard code (KC)
- actual key on keyboard (always lowercase)
- Unicode Character code (UCC)
- result Unicode character (
Shift + w = W, return the character code ofW)
| altKey | whether alt is pressed |
| ctrlKey | |
| shiftKey | |
| metaKey | whether meta is pressed. Windows key |
| key | key name on keyboard (F1,Enter..) |
| keyCode | keypress-0, keydown,keyup-UCC. All browsers. |
| charCode | keypress-UCC, keydown,keyup-0. All browsers. |
| which | keypress-UCC, keydown,keyup-UCC. All browsers. |
| location | 4 numbers 0 - standard,1 - left, |
| 2 - right (CTRL), 3 - numpad | |
- If you want to return UCC for all events cross all browsers use this
var x = event.charCode || event.keyCode;
7.1.5.9. HashChangeEvent Object
newURL :: the URL after the hash has been changed oldURL
7.1.5.10. PageTransitionEvent
For events: onpageshow and onpagehide persisted :: whether the page was cached
7.1.5.11. FocusEvent Object
relatedTarget :: the element related to the element that triggered the event event.relatedTarget.tagName;
7.1.5.12. AnimationEvent Object
animationName :: keyframe name elapsedTime :: number of seconds the transition has been running
7.1.5.13. HTML DOM Event
- DOMContentLoaded html:event:DOMContentLoaded
- Target can be
documentandwindow document.addEventListener('DOMContentLoaded', (e) => { ... });- (no term)
- It's fired when the initial HTML has been completely loaded and parsed, without waiting for stylesheets, images and subframes to finish loading. html:event:load is fully-loaded
- (no term)
- Synchronous JavaScript can issue a
doc.writeat any point; hence the DOM tree construction is blocked anytime a synchronous script is encountered - (no term)
- JavaScript can query for a computed style of any object, which means it can also block on CSS
- (no term)
- DOM construction can't proceed until JavaScript is executed, and JavaScript can’t proceed until CSSOM is available
- (no term)
- Mark JavaScript files as js:defer to unblock construction of DOM and execute them before
domContentLoadedEventStartbut JavaScript files still need to wait for CSSOM to be finished - (no term)
- Mark JavaScript files as js:async is similar to js:defer but DCL does not have to wait for execution of async scripts
- (no term)
- By default, JavaScript will block DOM construction, which may block on CSSOM. Sync scripts are bad, but you already knew that. Marking scripts with “defer” and “async” makes an implicit promise to the document parser that you will not use
doc.write, which in turn allows it to unblock DOM construction - (no term)
- If at any point we must wait for JavaScript execution, then we will have to first wait for the CSSOM construction to finish. In other words, there is a hard dependency edge between JavaScript and CSS… Stylesheets at the top, scripts at the bottom? Now you know why
- (no term)
- https://calendar.perfplanet.com/2012/deciphering-the-critical-rendering-path/
- Target can be
- load html:event:load
- Target is
window window.addEventListener('load', (e) => {})- (no term)
- Refer to html:event:DOMContentLoaded
Detailed events using
PerformanceNavigationTiming- domLoading
- browser is about to start parsing the first received bytes of the HTML document
- domInteractive
- browser has finished parsing all HTML and DOM construction
- domContentLoaded
- Both DOM is ready and there're no stylesheets that are blocking JavaScript execution (CSSOM is parsed and ready). Ready to construct the render tree.
- domContentLoadedEventStart
- domContentLoadedEventEnd
- domComplete
- all processing is complete and all resources e.g. images have finished downloading. The loading spinner has stopped spinning.
- load
- page is fully loaded
- loadEventStart
- loadEventEnd
- Target is
7.1.5.14. readystatechange html:event:readystatechange
- Refer to html:document:readyState
// Alternative to load event document.onreadystatechange = function () { if (document.readyState === 'complete') { initApplication(); } } document.addEventListener('readystatechange', event => { if (event.target.readyState === 'interactive') { initLoader(); } else if (event.target.readyState === 'complete') { initApp(); } });
7.1.5.15. MessageEvent
- https://developer.mozilla.org/en-US/docs/Web/API/MessageEvent
- Properties
- data
- e.g.
'http://localhost:8080'
7.1.6. DOMTokenList Object
A collection of space-separated tokens.
| DOMTokenList.methodName() | Notes |
|---|---|
| or | |
| DOMTokenList.propertyName | |
| length | read only |
| add("class2Name") | |
| remove("class2Name") | |
| replace("old","new") | |
| toggle('class2Name') | if exists, removes and return false. If not, add and return true |
| contains("class2Name") |
7.1.7. NamedNodeMap Object
A collection of Attr object 's. Loopable like an Array but no Array methods.
7.1.8. Attr Object
An attribute node.
| name | |
| value |
7.1.9. PointerEvent
7.2. Global Attributes
- Can be applied to any element
- https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes
- (no term)
idclassaccesskey- Add shortcut.
M-accesskey - download
- filename. force download
contenteditable- true or false. Become editable
contextmenu- right click menu. Only in Firefox
dirltrrtlorauto. text directionhiddenspellcheck- true or false
- title
- tool tip
7.2.2. Drag-and-Drop
ondrop ondragover draggable ondragstart
<!-- ondropover allows this tag to receive dropped elements. ondrop defines what to do with the traferred data --> <div id="div1" ondrop="drop(e)" ondragover="allowDrop(e)"></div> <!-- Make a tag draggable: draggable="true" <a> and <img> are by default draggable ondragstart defines what data to pass to ondrop --> <img id="drag1" src="http://placehold.it/300x250" draggable="true" ondragstart="drag(e)"> <script> function drag(e) { // Pass the dragged element id ev.dataTransfer.setData("text", ev.target.id); } function allowDrop(e) { e.preventDefault(); } function drop(e) { e.preventDefault(); var data = e.dataTransfer.getData("text"); e.target.appendChild(document.getElementById(data)); } </script>
7.2.3. tabindex
- Elements by default are focusable
- a with href
- Better having every a to have href attribute as even though adding tabindex attribute with a positive integer enables focus but cannot enter or spacebar to trigger onclick
- link with href
- button
- input that are not type="hidden"
- select
- textarea
- a with href
- MacOS
- By default, MacOS only tabs through
Text Boxes and list only, System Preferences > Keyboard Preferences- lists, text fields and textareas. No buttons and checkboxes
- By default, MacOS only tabs through
- Bringing an element to focus
- Hit tab on the keyboard
- Use JavaScript e.g.
document.getElementById('a').focus()
- integer
- -1 or any negative number
- make it unfocusable
- 0
- put it in the default focus order determined by its position in the HTML
- (no term)
- positive number
- Elements with positive tabindex will be placed in front of elements that don't have a tabindex attribute
- The smaller, the higher order starting from 1
- To test which is the first tab, click on the URL bar and then tab
7.3. Events and Event Attributes
- All DOM Events
- an event handler can have one of the 2 mechanisms:
- Capturing phase
- DOM heirarchy up
- Run outer-most ancestor's (
<html>) same event handler e.g.onclick - Then its child's same event handler, go on until the original element
- Run outer-most ancestor's (
- Bubbling phase (default)
- DOM heirarchy down
- Run the original element's same event handler
- Then its parent's same event handler, go on until the outer-most ancestor
- (no term)
- If an element has multiple event handlers of both mechanisms, the capturing phase will run first followed by bubbling
- (no term)
- There's another phase called target phase
- Which is activated for event handlers that are registered using
element.onclickor using HTML attribute. These event handlers are only triggered on the original elements, will not capture or bubble
- Which is activated for event handlers that are registered using
- These DOM or JavaScript objects have implemented
EventTargetinterface which can receive events and may have listeners- element, document, window
- js:xmlhttprequest, AudioNode, AudioContext and others
- Add an event
element.onsomeeventlowercasevar btn = document.getElementById('button-a'); btn.onclick = function() {} // anonymous function function bgCharge() {} btn.onclick = bgCharge; // named function // cannot have multiple event handlers
addEventListner()andremoveEventListener()https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener
btn.addEventListener('click', function() {}); btn.addEventListener('click', bgCharge); btn.removeEventListener('click', bgCharge); // only one of the identical event handlers can be registered // anonymous functions, even though they have the same code, they are considered different // target.addEventListener(type, listener [, options]); // options is an object // target.addEventListener(type, listener [, useCapture]); // useCapture default is false to use bubbling phase, true to capturing phase
Event delegation. e.g. in bubbling phase, add an event handler to the parent element of child elements, and this event handler knows which child element is clicked. This avoids adding unknown amount of event handlers to new/removed child elements
document.getElementById("myULelements").addEventListener("click",function(e) { // e.target was the clicked element if (e.target && e.target.matches("li.classA")) { console.log("Anchor element clicked!"); } });
- Event objects
refer to html:EventObject
function bgCharge(e) { e.target }
7.3.1. Windows Event in body tag
| onafterprint, onbeforeprint | only IE and FF |
| onbeforeunload | before page is closed/refreshed |
| onunload | after page is closed/refreshed |
| onhashchange | when URL hash is changed |
| onload | when an object is loaded: |
| body, frame, frameset, iframe, | |
| img, link, script, style | |
| Not run when it's loaded from cache | |
| onpageshow | similar to onload but always triggers |
| onpagehide | similar to onunload but onunload event causes |
| the page to not be cached. | |
| onresize | |
7.3.2. Form Event Attributes
Also can be applied on almost all elements.
| oncontextmenu | right click to bring context menu |
| onreset | |
| onsearch | <input type="search" onsearch="">. IE and FF not supported |
| onselect | when text is selected in <input type="text"> or <textarea> |
| onblur | leave a form field. opposite of onfocus |
| onfocus | opposite of onblur |
| onfocusin | similar to onfocus but bubbles. FF not supported |
| onfocusout | similar to onblur but bubbles.FF not supported |
| onchange | oninput and lose focus. also works on <select> and <keygen> |
| oninput | not working on <select> and <keygen> |
- oninput
<input>,<select>,<textarea>. Also works forcontenteditabledocument.designMode
7.3.3. Mouse Events, Clipboard Events, Keyboard Events
| ondblclick | double click |
| ondragstart | |
| ondrag | being dragged |
| ondragend | |
| ondragenter | when dragged element enters the drop target |
| ondragover | over the drop target |
| ondragleave | leave the drop target |
| ondrop | is dropped on the drop target |
| onscroll | |
| onwheel | only Chrome and FF |
| oncopy | |
| oncut | |
| onpaste | |
| onmousedown | |
| onmouseenter | when move into child element |
| onmouseleave | when moving out of the element (not its children) |
| onmousemove | when moving inside an element |
| onmouseover | when onto this element |
| onmouseout | when moving out of the element and its children |
| onkeydown | evens are in order. works for all keys |
| onkeypress | Some keys are not fired: C, A, S, ESC, etc. |
| onkeyup | when releasing a key |
7.3.4. Media Events
Elements: audio, img, embed, object, video Events in order:
| onloadstart | |
| ondurationchange | |
| onloadedmetadata | |
| onloadeddata | |
| onprogress | |
| oncanplay | |
| oncanplaytrhough | |
7.3.5. Animation Events
Refer to CSS @keyframes. animationstart animationend animationiteration :: when animaiton repeats
7.3.6. Transition Event
transitionend
7.3.7. Misc events
| onerror | when an error occurs while loading an external file: <img> |
7.4. Canvas
7.5. Tags
7.5.1. Tags
| Tag | Usage | Notes |
|---|---|---|
| a | download="filename" | force download |
| abbr | title="World Health Organization" | abbreviation WHO |
| address | author or owner (not just address) of a document | |
| or an article. display:block, italic author | ||
| base | href,target | base url and target for <a> tags |
| article | affect outline | |
| aside | ||
| section | headline tag is required | affect outline |
| nav | affect outline | |
| cite | <cite>U of T Professor</cite> | wrap a person's title |
| dfn | <dfn>HTML</dfn> is … | A term to be defined |
| mark | highlight text | background yellow and text in black |
| s | text-decoration: line-through | |
| q | <q cite="abc.com">Build</q> | Double quotes |
| ruby | <ruby>a <rt>sound</rt></ruby> | |
| wbr | <wbr>alongword</wbr> | the word will not be broken for line break |
| meter | min,max,high,low,optimum | value |
| progress | max, value | |
| colgrup, | specify styles and span for | |
| col | columns in a table | |
| iframe | srcdoc="" | IE & Edge not supported. Repalce double quote with " |
| sandbox="" | empty to apply all restrictions | |
| e.g. allow-forms allow-same-origin | ||
| input | autocomplete="on or off" | default on. turn off for a specific field |
| autofocus="" | ||
| formmethod="get or post" | overwrite <form> type="submit, image" | |
| formaction="another.php" | overwrite <form> type="submit, image" | |
| formenctype="multipart/form-data" | overwrite <form> type="submit, image" | |
| formtarget="_blank" | overwrite <form> type="submit, image" | |
| min="1" max="1979-12-31" | for type="date" or type="number" | |
| multiple | for type="file" | |
| pattern="regex" | for type=text,date,search,url,tel,email,password | |
| escape " in regex with \x22 | ||
| placeholder="hint" | ||
| step="3" | -3,0,3,6 can be accepted. type="number" | |
| type=email, url | ||
| required | no in IE | |
| type="range" min max | slide control | |
| type="search" | ||
| disabled="disabled" | not selectable, editable and not submitted | |
| readonly="readonly" | selectable, not editable and get submitted | |
| samp | sample computer output | |
| code | monospace | |
| kbd | monospace | |
| var | variable name | |
| li | value="100" | increment by 1 |
7.5.2. <a>
7.5.2.1. rel attribute
- Attribute value is called a keyword. Keywords are
- noreferrer
- For
<a>,<area>,<form> - Instruct the browser to omit the
Referrerheader when navigating to the target resource - Additionally, behave as if another keyword 1.2 is used
- For
- noopener
- For
<a>,<area>,<form> - Instruct the browser to not granting the new browsing context access when navigating to the target resource
Window.openerproperty on the opened window is null
- For
- noreferrer
7.5.3. <iframe> attribute sandbox
- Add space-separated
allow-*tokens as value insandboxattribute to lift particular restrictions. By default, all are not allowedallow-forms- form submission
allow-scripts
7.5.4. <picture>, <img srcset sizes>
7.5.4.1. html:img:srcset html:img:sizes
- IE 11 doesn't support
srcsetnorsizes - it's used to serve larger—but otherwise identical—image sources to high resolution displays only
- When
xis used insrcset,sizescannot be set <img srcset="examples/images/image-384.jpg 1x, examples/images/image-768.jpg 2x" alt="…">
- When
winsrcsetis usually the actual image's width- refer to wp:img:srcset:sizes
sizesattribute is the width that the image needs to have under matched the media condition- If
sizeshas media condition, only the first matched media condition from the left is chosen - If the image needs to take a third of the viewport, then
sizes="33.3vw" - Other than
px,vw, em, remcan also be used insizes. But not percentage - The
srcsetwhich has the closestwthat matches the chosensizeswill serve:- First choose
srcsetthat havewwhich is larger thansizes, among those choose the smallestsrcset - If no
srcsethaswthat is larger thansizes, choose the closest one (biggestsrcset) - e.g. 3
srcset: 100w, 200w, 300w and the chosensizesis 150px, then 200w is chosen. Ifsizesis 400px, then300wis chosen
- First choose
- If
In order to test img with srcset and sizes, you need to open InCognito and resize the window size not choosing a simulated device.
<img src="examples/images/fallback-smallest-size.jpg" sizes="(min-width: 40em) 80vw, 100vw" srcset="examples/images/medium.jpg 375w, examples/images/large.jpg 480w, examples/images/extralarge.jpg 768w" alt="…"> <style> /* make sure img tag is responsive */ img { max-width:100%; height:auto; } </style>
7.5.4.2. html:picture
Used when you need explicit control over which source is shown at set viewport sizes. Except IE 11 and below
<picture> <source srcset="a.jpg" media="(min-width: 600px)"> <source srcset="b.jpg" media="(min-width: 500px)"> <source srcset="fallback.jpg"> <img src="fallback.jpg" alt="one alt"> </picture> <!-- This is called art direction --> <picture> <source media="(min-width: 800px)" sizes="100vw" srcset="cropped-for-wide-screens--large.jpg 1600w, cropped-for-wide-screens--small.jpg 800w" /> <source media="(min-width: 600px)" sizes="100vw" srcset="full-image-for-standard-screens--large.jpg 1200w, full-image-for-standard-screens--small.jpg 600w" /> <img src="zoomed-in-for-small-screens--small.jpg" srcset="zoomed-in-for-small-screens--large.jpg 800w, zoomed-in-for-small-screens--small.jpg 400w" alt="" /> </picture> <picture> <source srcset='paul_irish.jxr' type='image/vnd.ms-photo'> <source srcset='paul_irish.jp2' type='image/jp2'> <source srcset='paul_irish.webp' type='image/webp'> <img src='paul_irish.jpg' alt='paul'> </picture>
7.5.4.3. srcset and sizes calculation
small is 500x100, medium is 1000x200, and large is 2000x400 Device with screen width of 320px and 1x display (non-retina) Then chooses the one that is closest to 1 500 / 320 = 1.5625 1000 / 320 = 3.125 2000 / 320 = 6.25
<img src="small.jpg" srcset="medium.jpg 1000w, large.jpg 2000w">
By default sizes="100vw", which is 100% viewport (320px in the above example)
Extra media query can be added
sizes="(max-width: 300px) 100vw, 300px"
Which means if media query matches, use 100vw. If not match, use 300px
Responsive image which takes 100% width. Define the largest width image first This is the WordPress way. wp:img:srcset:sizes
<img src="medium.jpg" width="1000" height="200" srcset="large.jpg 2000w, small.jpg 500w" sizes="(max-width: 1000px) 100vw, 1000px"> <style> /* make sure img tag is responsive */ img { max-width:100%; height:auto; } </style>
<picture> <source media="(max-width: 799px)" srcset="elva-480w-close-portrait.jpg"> <source media="(min-width: 800px)" srcset="elva-800w.jpg"> <img src="elva-800w.jpg" alt="Chris standing up holding his daughter Elva"> </picture>
CSS fallback
<img srcset=" examples/images/image-384.jpg 1x, examples/images/image-768.jpg 2x " alt="…"> <style> .img { background-image: url(examples/images/image-384.jpg); } @media (-webkit-min-device-pixel-ratio: 2), (min-resolution: 192dpi) { .img { background-image: url(examples/images/image-768.jpg); } } /* 1.25 dpr */ @media (-webkit-min-device-pixel-ratio: 1.25), (min-resolution: 120dpi){ /* Retina-specific stuff here */ } /* 1.3 dpr */ @media (-webkit-min-device-pixel-ratio: 1.3), (min-resolution: 124.8dpi){ /* Retina-specific stuff here */ } /* 1.5 dpr */ @media (-webkit-min-device-pixel-ratio: 1.5), (min-resolution: 144dpi){ /* Retina-specific stuff here */ } </style>
7.5.5. <dl>, <dt>, <dd> Definition Term/name in a Definition list
<dl> <dt>Term #1</dt> <dd>Description of Term #1</dd> <dt>Term #2</dt> <dd>Description 1 of Term #2</dd> <dd>Description 2 of Term #2</dd> </dl>
7.5.6. <blockquote> and <q>
<!-- blockquote and cite --> <blockquote> <p>...</p> <cite>- Jeremy Keith</cite> </blockquote> <!-- inline quote, it has different styles based on different language --> <p lang="fr"> Jeremy Keith said, <q>...</q> </p>
7.5.7. <time>
<time>May 8</time> <time datetime="2025-05-08">Tomorrow or any string</time> <time datetime="07:00">7:00 am</time> <time datetime="07:00:28.5">hh-mm-ss.ddd</time> <time datetime="15:45-05:00">hh-mm-ss.ddd[+-]hh:mm With timezone</time> <time datetime="2020-11-04T19:00">Wednesday, Nov 4th at 7pm</time> <time datetime="2020-11-04 19:00-0500">Wednesday, Nov 4th at 7pm</time>
7.5.8. <figure> <figcaption>
<figure> <img src="a.png" width="960" height="720" alt="shiny black dog in the sun"> <figcaption>Maggie the dog enjoys resting in a field, after a long day of chasing squirrels.</figcaption> </figure>
7.5.9. <video> html:video
- Codec
- a device or program that encodes or decodes a data stream or signal
- H.264
- H.263
- QuickTime Movie
- (no term)
- MP4 is the most compatible video format. Uses H.264 (not free, not open source) video codec and AAC audio codec
- (no term)
- WebM is Google for HTML5. Uses VP8 video codec and Vorbis audio codec. Doesn't support in iOS, Safari nor old IE
- (no term)
- FLV is Flash only. Uses H.263 video codec and MP3 audio codec
7.5.9.1. autoplay attribute
It's turned off on iOS mobile < v10
<video autoplay muted loop id="myVideo"> <source src="rain.mp4" type="video/mp4"> </video>
7.5.9.2. preload attribute html:video:preload
<video id="video" preload="metadata" src="file.mp4" controls></video>preloadoptions- metadata
- dimensions, track list, duration
- auto
- load entire video. Setting to auto is not guaranteed as some browsers force it to be metadata or none.
- (no term)
- none
- Video resource starts to be fetched after
DOMContentLoadedandwindow.loadevent will be fired when the entire resource is completely fetched - https://developers.google.com/web/fundamentals/media/fast-playback-with-video-preload
- Instead of using
<video preload="">, we can use html:link:rel:preloadPreload full small media files < 5mb
<link rel="preload" as="video" href="https://cdn.com/small-file.mp4"> <video id="video" controls></video> <script> // Later on, after some condition has been met, set video source to the // preloaded video URL. video.src = 'https://cdn.com/small-file.mp4'; video.play().then(_ => { // If preloaded video URL was already cached, playback started immediately. }); </script>
Preload the first segment use Media Source Extensions JavaScript API
<link rel="preload" as="fetch" href="https://cdn.com/file_1.webm"> <video id="video" controls></video> <script> const mediaSource = new MediaSource(); video.src = URL.createObjectURL(mediaSource); mediaSource.addEventListener('sourceopen', sourceOpen, { once: true }); function sourceOpen() { URL.revokeObjectURL(video.src); const sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp09.00.10.08"'); // If video is preloaded already, fetch will return immediately a response // from the browser cache (memory cache). Otherwise, it will perform a // regular network fetch. fetch('https://cdn.com/file_1.webm') .then(response => response.arrayBuffer()) .then(data => { // Append the data into the new sourceBuffer. sourceBuffer.appendBuffer(data); // TODO: Fetch file_2.webm when user starts playing video. }) .catch(error => { // TODO: Show "Video is not available" message to user. }); } </script>
7.5.9.3. Caption
- webvtt
- web video text tracks. Plain text with
.vttextension - (no term)
kindattribute- descriptions
- for blind people
- chapters
- allow video player to jump to a certain chapter
<video controls> <source src="moonwalk.480p.vp9.webm" type="video/webm"> <source src="moonwalk.480p.h264.mp4" type="video/mp4"> <track src="moonwalk.vtt" kind="captions" label="English" srclang="en" default> <track src="moonwalk.es-la.es.vtt" kind="subtitles" label="Español" srclang="es"> <p>This would be a video of a moonwalk, if your device supported playing this video.</p> </video>
7.5.10. <audio>
<audio controls loop autoplay> <source src="b.ogg" type="audio/ogg; codec=opus"> <source src="a.mp3" type="audio/mpeg"> Sorry your browser doesn't not support audio. </audio>
7.5.11. <tbody>, <thead> and <tfoot>
- All 3 are to group multiple rows
<tr>'s- To scroll
<tbody>independently of<thead>and<tfoot>
- To scroll
<tbody>- Parent is
<table> - Used to group rows
<tr>. Multiple groups are allowed - Use a
<tr>filled with<th>'s to create headers within each<tbody>- These
<th>'s have different browser native CSS styles
- These
- Parent is
<thead>and<tfoot>- Parent is
<table> - Only one is permitted in a
<table>
- Parent is
7.5.12. <input>
7.5.12.1. type=number
- When attribute
stepis supported, UI widget (arrow up/down keys and validation) is available- Not supported on mobile
- iOS Safari, Android browser, Chrome for Andriod
- Supported on desktop browsers
- for IE, Edge, FF, Chrome, Safari, Opera
- Attributes
step,min,maxare part of the validation and increment/decrementvalueis greater thanminand less thanmax
- UI widget happens only when a form is submitted. Validation cannot be triggered using JavaScript..
<input type="number"> <input type="number" step="0.01" class="currency"> <input type="number" step="0.01" min="0" class="currency-positive"> <input type="number" step="any" class="any-decimal">
7.5.12.2. type=range
- Supported on almost all browsers
min,max,step
7.5.12.3. type=email
- Only supported on modern mobile browsers
- Does not validate email at all. Use this
(val.replace(/\S+@\S+\.\S+/g, 'x') !== 'x' || val.value === 'x') ? 0 : 1
7.6. Meta
7.6.1. Meta Refresh (redirect)
<meta http-equiv="refresh" content="0;URL='http://abc.com'" /> <!-- "0" means zero second -->
7.6.2. Open Graph Protocol html:og
- http://ogp.me/
- https://developers.google.com/web/fundamentals/discovery/social-discovery/
- Required properties for every page
- og:title
- article, video, video.movie, etc. Refer to ogp types
- multiple
og:imageis possible - og:url
<meta property="og:title" content="" />- Refer to wp:plugin:wordpress-seo:og
7.6.3. Twitter Cards twitter:card
7.6.4. Address Bar html:address bar
Hide Safari UI (URL address bar)
<meta name="apple-mobile-web-app-capable" content="yes"> <!-- By default, iOS web content displays under the top black status bar. Push the content to the top and make the status transparent Default is black --> <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> <!-- Address Bar Color --> <meta name="theme-color" content="#4285f4">
7.6.5. Meta viewport
- Without this tag, the browser viewport will take the full resolution (window.screen.width * window.screen.devicePixelRatio)
- Not wise to use
min-device-widthandmax-device-widthin @media CSS
<meta name="viewport" content="width=device-width, minimum-scale=1, initial-scale=1, shrink-to-fit=no"> <!-- Set the width of the page (viewport) to follow the screen-width of the device (screen-width can be changed by resizing the browsing or changing the orientation) Initial zoom level 1.0 shrink-to-fit is Safari only Prevent zoom: add maximum-scale=1.0, user-scalable=0 the above won't be WCAG compliant -->
7.6.6. <meta charset="utf-8">
7.6.7. Response header html:meta:http-equiv:response header
HTTP response header can be defined in html markup
<meta http-equiv="Content-Security-Policy" content="default-src https://cdn.example.net; child-src 'none'; object-src 'none'">
7.7. Header
- All headers
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
- Response header in html meta
- html:meta:http-equiv:response header
7.7.1. Age header:age
A non-negative integer of seconds that the object (response) has been in a proxy cache (CDN, Varnish)
7.7.2. Cache-Control, Pragma - response header:cache-control
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
- Refer to apache:mod_headers:cache
Pragmais for legacy HTTP/1.0 whereCache-Control HTTP/1.1is not yet present. OnlyPragma:no-cacheorPragma:cache- Directives
- must-revalidate
- force to verify the status before using the cache and expired ones should not be used.
- no-cache
- browser has to always check Etag and makes a roundtrip request. If file in server is new, it will be downloaded.
- no-store
- no cache should be stored in request or response e.g. Go Back to previous page, the browser shows the browser cache version of the page
- no-transform
- No transformations or conversions should be made to the resource. The
Content-Encoding,Content-Range,Content-Typeheaders must not be modified by a proxy. A non-transparent proxy might, for example, convert between image formats in order to save cache space or to reduce the amount of traffic on a slow link. Theno-transformdirective disallows this. - public
- response can be cached in all devices. It's not necessary as max-age inidicates the response is cacheable
- private
- response can be cached for a single user (user's browser) but not in shared cache (public CDN, Varnish or other intermediaries)
- proxy-revalidate
- max-age=<seconds>
- refer to header:expires If max-age is set in Cache-Control and Etag is set, then after max-age, client sends a request with If-None-Match: <etag-value> to server and if the Etag response doesn't change, server will return 304 Not Modified response and client will not download the file. The cache serves the same file for another max-age before checking Etag again
post-check=0, pre-check=0- (Discouraged) Old IE does not cache and always ask from server
- Common usage
Cache-Control: no-cache, max-age=315360000- serve the cache for 1 year but if server version has updated, download from server
public, max-age=2592000- Tells the
externalcaching system e.g. Varnish not to make a call to the web server for this request until the max-age header:cache-control:public max-age no-cache, must-revalidate, post-check=0, pre-check=0- no cache and always ask from server
max-age=0- Chrome adds this in request when you enter the URL or hit refresh. This tells the cache server to serve the cache whenever possible. Whether or not the cache system serves the cache depends on the cache server. e.g. Varnish still serves the cache when it's possible
7.7.3. Expires header:expires
When the response is considered stale. Set to 0 means the response is expired. If Cache-Control has max-age or s-max-age, Expires is ignored.
7.7.4. Content-Security-Policy (CSP) - response
Content-Security-Policyis official name- X-Content-Security-Policy (Old Firefox and IE)
- X-WebKit-CSP (Chrome and Safari)
- https://developers.google.com/web/fundamentals/security/csp/
- use Content-Security-Policy-Report-Only instead of Content-Security-Policy to test first!
A directive is to specify whitelist
Content-Security-Policy: script-src 'self' https://apis.google.com https://host1.com https://host2.com
- is any other non-defined directives' default
Define multiple directives,
'none'means whitelist nothing (block anything)Content-Security-Policy: default-src https://cdn.example.net; child-src 'none'; object-src 'none'
*://*.example.com:*data:http:script-src * data: https://ssl.gstatic.com 'unsafe-inline' 'unsafe-eval';
example.commeans any scheme and any port- matches the current origin, but not its subdomains
- allow inline JavaScript and CSS
- allow text-to-JavaScript mechanisms like eval
By default, CSP bans inline script entirely if 'unsafe-inline', 'nonce-*' or 'sha256-' is not used
- scripts inside <script> tags
- inline event handlers and
javascript:URLs
You will need to move content of script tags into an external file:
<!-- change this --> <script> function doAmazingThings() { alert('YOU AM AMAZING!'); } </script> <button onclick='doAmazingThings();'>Am I amazing?</button> <!-- To this --> <!-- amazing.html --> <script src='amazing.js'></script> <button id='amazing'>Am I amazing?</button>
amazing.js
function doAmazingThings() { alert('YOU AM AMAZING!'); } document.addEventListener('DOMContentReady', function () { document.getElementById('amazing') .addEventListener('click', doAmazingThings); });
report-uri directive tells the browser to POST JSON-formatted violation reports
Content-Security-Policy: default-src 'self'; ...; report-uri /my_amazing_csp_report_parser;
// example
{
"csp-report": {
"document-uri": "http://example.org/page.html",
"referrer": "http://evil.example.com/",
"blocked-uri": "http://evil.example.com/evil.js",
"violated-directive": "script-src 'self' https://apis.google.com",
"original-policy": "script-src 'self' https://apis.google.com; report-uri http://example.org/my_amazing_csp_report_parser"
}
}
Generate nonce value differently each time and insert it into any inline script, bash64
<script nonce=EDNnf03nceIOfn39fn3e9h3sdfa> //Some inline code I cant remove yet, but need to asap. </script> Content-Security-Policy: script-src 'nonce-EDNnf03nceIOfn39fn3e9h3sdfa'
Or use sha
<script>alert('Hello, world.');</script>
Content-Security-Policy: script-src 'sha256-qznLcsROx4GACP2dm0UCKCzCG-HiZ1guq6ZZDob_Tng='
// calculate sha hash for each inline script
To enable Google Analytics without 'unsafe-inline', move GA code in a js file and
Content-Security-Policy: script-src www.google-analytics.com; img-src www.google-analytics.com
I've heard there's way other than 'unsafe-inline' for Google Tag Manager
referrer directive is like header:referrer-policy
7.7.5. X-Frame-Options
Whether the client can render the page in a <frame>, <iframe>, <object>.
DENY SAMEORIGIN :: the page can only be displayed in a frame on the same origin as the page itself. ALLOW-FROM uri :: Chrome doesn't support.
7.7.6. Content-Type
Content-Type: text/html; charset=UTF-8- Directive
boundaryis required for rfc:mime:multipart/form-data - Refer to rfc:mime
7.7.7. X-Content-Type-Options - response
- Only one value
X-Content-Type-Options: nosniff- (no term)
- It only applies to script and style
- (no term)
- Blocks a request if the requested type is
styleand MIME type is not "text/css"scriptand MIME type is not JavaScript MIME type
7.7.8. Content-Disposition
- When used in email, it is more powerful. But in HTTP context, it has only 3 usage
Content-Disposition: inline- safe to be displayed in or as Web page
- (no term)
Content-Disposition: attachment- or
Content-Disposition: attachment; filename="filename.jpg" - Tells browser to download e.g. pop up `Save as` dialog
- or
- (no term)
Content-Disposition: form-data; name="fieldName"- or
Content-Disposition: form-data; name="fieldName"; filename="filename.jpg" - Refer to rfc:mime:multipart/form-data
- or
- php:rawurlencode
7.7.9. X-XSS-Protection - response
Not supported in FireFox. Use Content-Security-Policy for modern browsers. This is used for older browsers.
- Disable XSS filtering
X-XSS-Protection: 0- Default, If XSS then the client will sanitize the page (remove the unsafe parts)
X-XSS-Protection: 1- Rather than sanitizing the page, the client will prevent rendering of the page if an attack is detected
X-XSS-Protection: 1; mode=block- Chrome only. If XSS is detected, the client will sanitize the page and report the violation. Same as report-uri in Content-Security-Policy
X-XSS-Protection: 1; report=<reporting-uri>
7.7.10. Referer - request header:referer
7.7.11. Referrer-Policy - response header:referrer-policy
Only works for Chrome, FireFox and Opera Controls how the client sends the Referer request header with requests that are made from your site. e.g. clicking on a link on your website and the link goes to either your website or other websites.
no-referrer-when-downgrade :: If no Referrer-Police is defined, clients should set to this. The url is sent as a referrer when the protocol security level stays the same HTTP->HTTP or HTTPS->HTTPS
- source:http://a.ca/1/ Destination:http://b.com Referer is http://a.ca/1/
- source:http://a.ca/1/ Destination:https://b.com Referer is http://a.ca/1
- source:https://a.ca/1/ Destination:http://b.com Referer is NULL
- source:https://a.ca/1/ Destination:http://a.ca/2/ Referer is NULL
- source:https://a.ca/1/ Destination:https://a.ca/2/ Referer is https://a.ca/1/
no-referrer :: never set Referer header
same-origin :: only set Referer when the origin is the same.
- source:https://a.ca/1/ Destination:https://a.ca/2/ Referer is https://a.ca/1/
- source:https://a.ca/1/ Destination:http://a.ca/2/ Referer is NULL
origin :: always set the Referer header to the origin (domain) from which the request was made. Path information is stripped.
strict-origin :: This value is similar to origin above but will not allow the secure origin to be sent on a HTTP request, only HTTPS.
origin-when-cross-origin :: The browser will send the full URL to requests to the same origin/domain but only send the origin when requests are cross-origin.
strict-origin-when-cross-origin :: Similar to origin-when-cross-origin above but will not allow any information to be sent when a scheme downgrade happens (the user is navigating from HTTPS to HTTP).
unsafe-url :: The browser will always send the full URL with any request to any origin.
7.7.12. Strict-Transport-Security - response
- HSTS lets a website tell browsers that it should only be accessed using HTTPS. The first time your site is accessed using HTTPS and it returns the Strict-Transport-Security header, the browser records this information, so that future attempts to load the site using HTTP will automatically use HTTPS instead
Strict-Transport-Security: max-age=<expire-time>- max-age
- The time, in seconds, that the browser should remember that a site is only to be accessed using HTTPS
Strict-Transport-Security: max-age=<expire-time>; includeSubDomains- includeSubDomains Optional
- If this optional parameter is specified, this rule applies to all of the site's subdomains as well
Strict-Transport-Security: max-age=<expire-time>; preload- preload requires to register the domain on https://hstspreload.org
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security#Preloading_Strict_Transport_Security
- Applied in hosting:pantheon:hsts
7.7.13. Vary - response header:vary
Vary: <header-name>, <header-name>- Tell caching servers to deliver a different cache for different User-Agent
- Tell caching servers to always deliver a refresh copy (all requests are uncacheable)
- tell caching servers to deliver a different cache for different cookie header
7.7.14. Link header:link
- Each link defined in HTML is separated by comma and values inside a link with attributes are delimited by
; - Refer to html:link
7.7.14.1. html:hreflang
7.7.15. Accept-CH - response header:accept-ch
- Client Hints
- Tell browsers to give the following headers for following requests:
- DPR
- Viewport-Width
- actual width of an image in real physical pixels
- client max download speed
- boolean whether extra measures should be taken to reduce the payload
Initial page load has this in HTML
<img src="flower.jpg" sizes="25vw">
sizes attribute has to be set in order for the Width header to be sent. In the near future, the width attribute on the image element is likely to be included in the algorithm too, but for now, we’ll have to stick with sizes. The sizes attribute describes the layout and display size of the image.
- Header Width is then perfectly calculated.
- Header DPR saves x in srcset and thus saves some markup.
- deliver images which have approximate size
- Use image server/proxy to deliver pin-point size picture
- Base
http://example.com/image.jpg - Actual url in HTML
http://[key].lite.imgeng.in/http://example.com/image.jpg - Image server will deliver the images based on the CH request headers
- Base
7.7.16. X-Powered-By
7.8. MIME
- Media type also known as Multipurpose Internet Mail Extensiions or MIME
- https://www.iana.org/assignments/media-types/media-types.xhtml
- rfc:mime:multipart/form-data
- application/octet-stream
- With header:content-disposition:attachment, browsers propose `Save as` dialog to download
- Other
application/*, browsers may propose to open with a matching software app/program - Default for binary files also means unknown binary file
PHP force a file to download
$fileNameAfterDownload = 'a.pdf'; $size = filesize($fn); header('Content-Type: application/octet-stream'); header("Content-disposition: attachment; filename=\"{$fileNameAfterDownload}\""); // Maybe unnecessary header('Content-Transfer-Encoding:binary'); header("Content-Length:" . filesize($fn)); header("Expires:0"); header("Cache-Control:no-cache, must-revalidate"); header("Pragma:no-cache"); // This is to make sure we get a refresh copy of the source file // e.g. if file_get_contents($fn) is run before this, and the file is modified after the 1st file_get_contents and before this, the 2nd file_get_contents in here gets the cached copy clearstatcache(); $chunkSize = 25*(1024 * 1024); //25mb per chunky chunk if ($size > $chunkSize) { $handle = fopen($fn, 'rb'); while (!feof($handle)) { print(@fread($handle, $chunkSize)); ob_flush(); flush(); } fclose($handle); } else { echo file_get_contents($fn); } die();
7.9. Cookie
HTTP response header
Set-Cookie: <name>=<value>[; <Max-Age>=<age>] [; expires=<date>][; domain=<domain_name>] [; path=<some_path>][; secure][; HttpOnly]
httponly- client side script cannot access the cookie
secure- the client browser will prevent the transmission of a cookie over an unencrypted channel (non-https)
- (no term)
SameSite- Same Site
www.a.cais eq. toa.ca, and froma.cato any subdomains is the same site- while from
b.a.catoc.a.cais not same site
- only action e.g.
<a>links, cookies will be passed. Default since Feb 2020 for Chrome - requires
secure
- Same Site
- (no term)
- Refer to setcookie, php:ini:session:cookie_secure, apache:mod_headers for adding flags to cookies
- (no term)
- A single cookie can hold less than 4KB. A lot of browsers restrict max 20 cookies for a domain
- (no term)
- Session relies on cookie usually
sessionid(PHP default is to usePHPSESSIDas cookie)- Session stores data on server side and hence can hold a lot of data
- (no term)
- Token vs session
- token is stored on client side but generated by servers. Received by servers to validate permissions
- session relies on both the client side's (JS, HTML, cookie) is served from the same server
- while token can be used when front end and back end are served separately
7.10. Email
7.10.1. View email source on Outlook
Open the email, in Move tab under Message, Actions > Other Actions > View Source
7.10.2. Doctype
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <meta http-equiv="X-UA-Compatible" content="IE=edge" /> <meta name="robots" content="noindex, nofollow"> <link href='https://fonts.googleapis.com/css?family=Montserrat:400,700' rel='stylesheet' type='text/css'> <title>Product Update Hybrid</title> <style type="text/css"> @import url(https://fonts.googleapis.com/css?family=Montserrat); body, table, td, a{-webkit-text-size-adjust: 100%; -ms-text-size-adjust: 100%;} table, td{mso-table-lspace: 0pt; mso-table-rspace: 0pt;} img{border: 0; height: auto; line-height: 100%; outline: none; text-decoration: none; -ms-interpolation-mode: bicubic;} table{border-collapse: collapse !important;} body{font-family: 'Montserrat', Arial, sans-serif; height: 100% !important; margin: 0 !important; padding: 0 !important; width: 100% !important;} /* https://github.com/seanpowell/Email-Boilerplate */ div[style*="margin: 16px 0;"] { margin:0 !important; } a[x-apple-data-detectors] { color: inherit !important; text-decoration: none !important; font-size: inherit !important; font-family: inherit !important; font-weight: inherit !important; line-height: inherit !important; } </style> <!--[if mso]> <style type="text/css"> .body-text { font-family: Arial, sans-serif !important; } </style> <![endif]--> </head>
https://litmus.com/community/learning/24-how-to-code-a-responsive-email-from-scratch Use XHTML 1.0 Transitional
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<meta name="format-detection" content="telephone=no">
<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=no;">
<meta http-equiv="X-UA-Compatible" content="IE=9; IE=8; IE=7; IE=EDGE" />
<title>Page title</title>
<style type="text/css">
@media screen and (max-width: 630px) {
}
</style>
</head>
<body style="padding:0; margin:0">
<table border="0" cellpadding="0" cellspacing="0" style="margin: 0; padding: 0" width="100%">
<tr>
<td align="center" valign="top">
</td>
</tr>
</table>
</body>
</html>
Style
<style> @import url(http://fonts.googleapis.com/css?family=Roboto:300); /*Calling our web font*/ /* Some resets and issue fixes */ body, table, td, a{-webkit-text-size-adjust: 100%; -ms-text-size-adjust: 100%;} table, td{mso-table-lspace: 0pt; mso-table-rspace: 0pt;} img{border: 0; height: auto; line-height: 100%; outline: none; text-decoration: none; -ms-interpolation-mode: bicubic;} table{border-collapse: collapse !important;} body{font-family: 'Montserrat', Arial, sans-serif; height: 100% !important; margin: 0 !important; padding: 0 !important; width: 100% !important;} #outlook a{ padding:0; } .ReadMsgBody{ width:100%; } .ExternalClass{ width:100%; } .backgroundTable{ margin:0 auto; padding:0; width:100%; !important; } .ExternalClass *{ line-height:115%; } /* End reset */ /* These are our tablet/medium screen media queries */ @media screen and (max-width:630px){ /* Display block allows us to stack elements */ *[class="mobile-column"]{ display:block; } /* Some more stacking elements */ *[class="mob-column"]{ float:none !important; width:100% !important; } /* Hide stuff */ *[class="hide"]{ display:none !important; } /* This sets elements to 100% width and fixes the height issues too, a god send */ *[class="100p"]{ width:100% !important; height:auto !important; } /* For the 2x2 stack */ *[class="condensed"]{ padding-bottom:40px !important; display:block; } /* Centers content on mobile */ *[class="center"]{ text-align:center !important; width:100% !important; height:auto !important; } /* 100percent width section with 20px padding */ *[class="100pad"]{ width:100% !important; padding:20px; } /* 100percent width section with 20px padding left & right */ *[class="100padleftright"]{ width:100% !important; padding:0 20px 0 20px; } /* 100percent width section with 20px padding top & bottom */ *[class="100padtopbottom"]{ width:100% !important; padding:20px 0px 20px 0px; } } </style>
7.10.3. Guidelines
Campaign Monitor Mobile Design Guide
| Element | Note |
|---|---|
| table | always assign align, width,border, cellpadding, cellspacing |
| tr | <tr style="padding:0;margin:0;line-height:0;" height="15"> |
| <td> </td> | |
| </tr> // empty row with height | |
| td | Wrap text in td |
| Attribute | |
| style | use display: none!important; |
Inline text full width separator. Change attribute size and style height when needed.
<hr noshade color="#FFFFFF" width="100%" size="1"
style="padding:0; margin:8px 0 8px 0; border:none; width:100%; height: 1px; color:#FFFFFF; background-color: #FFFFFF" />
<h2 style="font-family: 'Montserrat', Arial, sans-serif; font-size:20px; line-height:26px; color:#222222; font-weight:bold; text-transform:uppercase; padding:0 20px; margin:0;">You've been invited to this years event</h2>
7.10.4. Responsive
<style>
.sectionContainer {
background-color: #ffffff;
padding-right:10px !important;
padding-left:10px !important;
}
.sectionTable {
background-color: #ffffff;
}
@media only screen and (max-width:640px) {
.sectionTable {
max-width:640px !important;
width:100% !important;
}
}
@media only screen and (max-width:480px) {
.sectionTable {
max-width:480px !important;
width:100% !important;
}
.sectionContent .s480-w0 {
display:none !important;
}
}
</style>
<table>
<tr>
<td class="sectionContainer">
<table border="0" cellpadding="0" cellspacing="0" width="640" class="sectionTable">
<tr style="padding:0;margin:0;line-height:0;" height="15"><td> </td></tr>
<tr>
<td align="center" valign="middle" style="background-color:#004663;color:#ffffff;margin:0;font-size:24px;font-family:Georgia;font-weight:normal;">
Section #1
</td>
</tr>
<tr style="padding:0;margin:0;line-height:0;" height="10"><td> </td></tr>
<tr>
<td align="center" valign="top">
</td>
</tr>
</table>
<td>
</tr>
</table>
7.10.5. Even 3 columns
<td align="center" valign="top" style="font-size:0;">
<!--[if (gte mso 9)|(IE)]>
<table align="center" border="0" cellspacing="0" cellpadding="0" width="600">
<tr>
<td align="left" valign="top" width="190">
<![endif]-->
<div style="display:inline-block; max-width:33.3333%; min-width:190px; vertical-align:top; width:100%;" class="mobile-wrapper">
<table align="left" border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width:190px;" class="max-width">
<tbody><tr>
<td align="center" valign="top" style="font-family: Open Sans, Helvetica, Arial, sans-serif; padding-top: 25px;">
<img src="icon-shield.png" width="50" height="50" border="0" style="display: block;">
<h3 style="font-size: 18px; line-height: 24px;">Gmail</h3>
<p style="color: #999999; font-size: 14px; line-height: 20px;">
Mandeville carneiro robbins goas ross kelly ragan rodriguez stig jordan hodgekiss merlin yeaman
<br><br>
<a href="http://litmus.com" target="_blank" style="text-decoration: none; color: #75b6c9;">Read more →</a>
</p>
</td>
</tr>
</tbody></table>
</div>
<!--[if (gte mso 9)|(IE)]>
</td>
<td width="15" style="font-size: 1px;"> </td>
<td align="left" valign="top" width="190">
<![endif]-->
<div style="display:inline-block; max-width:33.3333%; min-width:190px; vertical-align:top; width:100%;" class="mobile-wrapper">
<table align="center" border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width:190px;" class="max-width">
<tbody><tr>
<td align="center" valign="top" style="font-family: Open Sans, Helvetica, Arial, sans-serif; padding-top: 25px;">
<img src="icon-cloud-lock.png" width="50" height="50" border="0" style="display: block;">
<h3 style="font-size: 18px; line-height: 24px;">Outlook</h3>
<p style="color: #999999; font-size: 14px; line-height: 20px;">
Mandeville carneiro robbins goas ross kelly ragan rodriguez stig jordan hodgekiss merlin yeaman
<br><br>
<a href="http://litmus.com" target="_blank" style="text-decoration: none; color: #75b6c9;">Read more →</a>
</p>
</td>
</tr>
</tbody></table>
</div>
<!--[if (gte mso 9)|(IE)]>
</td>
<td width="15" style="font-size: 1px;"> </td>
<td align="left" valign="top" width="190">
<![endif]-->
<div style="display:inline-block; max-width:33.3333%; min-width:190px; vertical-align:top; width:100%;" class="mobile-wrapper">
<table align="right" border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width:190px; float: right;" class="max-width">
<tbody><tr>
<td align="center" valign="top" style="font-family: Open Sans, Helvetica, Arial, sans-serif; padding-top: 25px;">
<img src="icon-key.png" width="50" height="50" border="0" style="display: block;">
<h3 style="font-size: 18px; line-height: 24px;">Lotus</h3>
<p style="color: #999999; font-size: 14px; line-height: 20px;">
Mandeville carneiro robbins goas ross kelly ragan rodriguez stig jordan hodgekiss merlin yeaman
<br><br>
<a href="http://litmus.com" target="_blank" style="text-decoration: none; color: #75b6c9;">Read more →</a>
</p>
</td>
</tr>
</tbody></table>
</div>
<!--[if (gte mso 9)|(IE)]>
</td>
</tr>
</table>
<![endif]-->
</td>
7.10.6. My even 2 columns
@media only screen and (max-width:480px){
td[class="articleBlockRightColumn"]{
text-align:center !important;
}
td[class="articleBlockLeftColumn"],
td[class="articleBlockRightColumn"]{
display:block !important;
width:100% !important;
padding-bottom:10px;
}
}
$html_default = [
'left' => 'articleBlockLeftColumn',
'right' => 'articleBlockRightColumn',
'ad' => 'bigAds',
];
?>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr>
<td align="left" valign="top" width="50%" class="<?php echo $html['left']; ?>">
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr>
<td align="left" valign="top">
<h3>
<a href="<?php echo $item['url']; ?>" target="_blank" style="text-decoration: none;">
<?php
print theme('nc_custom_templates_inline_native_article',
['article' => $item]
);
echo $item['title'];?></a>
</h3>
<p><?php echo $item['summary']; ?> <?php
if ($readmore['inline']) {
print theme(
'nc_custom_templates_inline_read_more',
['item' => $item,
'options'=> $readmore
]);
}
?></p>
<?php
if (!$readmore['inline']) {
print theme(
'nc_custom_templates_inline_read_more',
['item' => $item,
'options'=> $readmore
]);
}
?>
</td>
</tr>
</table>
</td>
<td align="right" valign="top" width="50%" class="<?php echo $html['right']; ?>">
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr>
<td align="right" valign="top" class="<?php echo $html['ad']; ?>">
<a href="<?php echo $dfp['a']; ?>">
<img src="<?php echo $dfp['img']; ?>" />
</a>
</td>
</tr>
</table>
</td>
</tr>
<tr><td> </td></tr>
</table>
7.10.7. My template
Presets
<?php
$_css_body = '-webkit-text-size-adjust:100%; -ms-text-size-adjust:100%;width:100% !important; margin:0;padding:0';
$_css_preheader = 'display: none; font-size: 1px; line-height: 1px; max-height: 0px; max-width: 0px; opacity: 0; overflow: hidden;';
$_css_section_column = 'background-color:#ffffff;padding-right:10px !important;padding-left:10px !important;';
// It's ok to set td with padding-right and left if td will not display:block when it's in mobile
// If td will display:block, then wrap content with <table><tr><td style="padding: 1em"></td></tr></table> to set padding.
$_css_section_table = 'max-width:640px;';
$_css_section_top = 'padding-top:15px;';
$_css_text_small = 'font-size:10px;color:#555555;font-family:Arial,Helvetica,sans-serif !important;';
$_css_p = 'font-size:16px;line-height: 20px !important;margin: 1em 0;font-family: Arial, Helvetica, sans-serif;';
$_css_h1 = 'font-family:Arial,Helvetica,sans-serif !important; font-size:30px; line-height:36px; font-weight:bold; color:#ffffff;padding:0; margin:0;';
$_css_h2 = 'font-family:Arial,Helvetica,sans-serif !important;font-size:20px;line-height:26px; font-weight:bold; color:#ffffff;padding:0; margin:0;';
$_css_h3 = 'font-family:Arial,Helvetica,sans-serif !important;font-size:16px;line-height:22px;font-weight:bold;color:#333333;padding:0; margin:0;';
$_css_h3_a = 'font-family:Arial,Helvetica,sans-serif !important;font-size:16px;line-height:22px;font-weight:bold;color:#333333;padding:0; margin:0;text-decoration: none;';
$_css_h4 = 'font-family:Arial,Helvetica,sans-serif !important;font-size:14px;line-height:20px;font-weight:bold;color:#333333;padding:0; margin:0;';
$_css_h4_a = 'font-family:Arial,Helvetica,sans-serif !important;font-size:14px;line-height:20px;font-weight:bold;color:#333333;padding:0; margin:0;text-decoration: none;';
?>
<style type="text/css">
td[class="articleBlockFullColumn"] {
padding-bottom:10px !important;
}
@media screen and (max-width:640px) {
}
@media only screen and (max-width: 480px) {
p {
padding-left: 0px !important;
padding-right: 10px !important;
}
td[class="mobilecenter"] {
text-align: center !important;
}
td[class="articleImg"],
td[class="articleDiv"] {
display:none;
}
}
</style>
Global
<body style="margin:0;padding:0;">
<table width="100%" height="100%" cellpadding="0" cellspacing="0" border="0" bgcolor="#F1F1F1">
<tr>
<td width="100%" valign="top" align="center">
<div style="<?php echo $_css_preheader; ?>"><?php echo $_preview_text; ?></div>
<center>
<table width="100%" height="100%" cellpadding="0" cellspacing="0" border="0">
<tr><td>Section 1</td></tr>
</table>
</center>
</td>
</tr>
</table>
</body>
Per section
<tr>
<td align="center" valign="top" style="<?php echo $_css_section_column; ?>">
<table border="0" cellpadding="0" cellspacing="0" width="100%" style="<?php echo $_css_section_table; ?>">
<tr>
<td align="center" valign="top" style="<?php echo $_css_section_top; ?>">
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<?php // section title ?>
<tr>
<td align="left" valign="top">
<h2 style="<?php echo $_css_section_title; ?>">Press Releases</h2>
<hr noshade color="#eb1f2a" width="100%" size="1"
style="padding:0; margin:8px 0 8px 0; border:none; width:100%; height: 1px; color:#eb1f2a; background-color: #eb1f2a" />
</td>
</tr>
<?php // section content ?>
<tr>
<td align="left" valign="top">
<?php for ($i = 0; $i < count($_item); $i++) {
$content_item = $_item[$i];
print theme(
'nc_custom_templates_component_full_width',
[
'item' => $content_item,
'readmore' => $readmore,
'html' => $full_width_html,
]
);
}
?>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
Section content
<?php
/**
* Created by PhpStorm.
* User: Li Li
* Date: 8/3/2017 2:28 PM
*
* $html => [
* 'row' => 'articleBlockFullColumn',
* 'img' => [
* 'class' => 'articleImg'],
* 'ad' => 'bigAds',
* ]
*/
$html_default = [
'row' => 'articleBlockFullColumn',
'h3' => 'color:#333333;font-weight:bold;font-size:20px;font-family:Arial,Helvetica,sans-serif !important;',
'h3 a' => 'text-decoration: none;color:#333333',
'p' => 'line-height: 20px !important;margin: 1em 0;font-family: Arial, Helvetica, sans-serif;',
'img' => [
'enable' => 1, // To show article thumbnail if it has one?
'class' => 'articleImg',
'w' => 140,
'h' => 105,
'style' => 'width:140px; height:105px; vertical-align:top;',
],
'div' => [
'class' => 'articleDiv',
'w' => 15,
],
];
?>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr>
<td align="left" valign="top" width="100%" class="<?php echo $html['row']; ?>">
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr>
<?php if (!empty($item['image']) && $html['img']['enable']): ?>
<td class="<?php echo $html['img']['class']; ?>" align="center" valign="top">
<a
href="<?php echo $item['url']; ?>"
target="_blank"><img alt=""
src="<?php echo $item['image']; ?>"
width="<?php echo $html['img']['w']; ?>"
height="<?php echo $html['img']['h']; ?>"
style="<?php echo $html['img']['style']; ?>"/></a>
</td>
<td class="<?php echo $html['div']['class']; ?>"
width="<?php echo $html['div']['w']; ?>"> </td>
<?php endif; ?>
<td align="left" valign="top">
<h3 style="<?php echo $html['h3']; ?>"><a
href="<?php echo $item['url']; ?>"
target="_blank"
style="<?php echo $html['h3 a']; ?>"><?php
print theme('nc_custom_templates_inline_native_article',
['article' => $item]
);
echo $item['title'];
?></a></h3>
<p style="<?php echo $html['p']; ?>"><?php echo $item['summary']; ?> <?php
if ($readmore['inline']) {
print theme(
'nc_custom_templates_inline_read_more',
['item' => $item,
'options'=> $readmore
]);
}
?>
</p>
<?php
if (!$readmore['inline']) {
print theme(
'nc_custom_templates_inline_read_more',
['item' => $item,
'options'=> $readmore
]);
}
?>
</td>
</tr>
</table>
</td>
</tr>
<tr><td> </td></tr>
</table>
Table with 1 col wrapper
$h_default = [
'spacer' => '', // '' no spacer, top or bottom
'content' => '',
'align' => ['','right'], // first one is table then td
'valign' => ['','top'],
'class' => ['','mobilecenter'],
'style' => ['',''],
];
?>
<table border="0" cellpadding="0" cellspacing="0" width="100%"
<?php if ($h['align'][0]) {echo ' align="'.$h['align'][0].'"'; } ?>
<?php if ($h['valign'][0]) {echo ' valign="'.$h['valign'][0].'"'; } ?>
<?php if ($h['class'][0]) {echo ' class="'.$h['class'][0].'"'; } ?>
<?php if ($h['style'][0]) {echo ' style="'.$h['style'][0].'"'; } ?>>
<?php if (isset($h['spacer']) && $h['spacer'] == 'top'): ?>
<tr><td> </td></tr>
<?php endif; ?>
<tr>
<td <?php if ($h['align'][1]) {echo ' align="'.$h['align'][1].'"'; } ?>
<?php if ($h['valign'][1]) {echo ' valign="'.$h['valign'][1].'"'; } ?>
<?php if ($h['class'][1]) {echo ' class="'.$h['class'][1].'"'; } ?>
<?php if ($h['style'][1]) {echo ' style="'.$h['style'][1].'"'; } ?>><?php echo $h['content']; ?></td>
</tr>
<?php if (isset($h['spacer']) && $h['spacer'] == 'bottom'): ?>
<tr><td> </td></tr>
<?php endif; ?>
</table>
Table with 2 cols
$_partnership_left = theme(
'nc_custom_templates_component_col_1',
[
'h' => [
'spacer' => '',
// '' no spacer, top or bottom
'content' => "<a href='${_dfp_txt_1['href']}' target='_blank' style='ouline:none;display:inline-block;'><img src='${_dfp_txt_1['img']}' alt='' style='${_dfp_txt_1['img_style']}'></a>",
'align' => [ '', 'center' ],
// first one is table then td
'valign' => [ '', 'top' ],
'class' => [ '', 'mobilecenter' ],
'style' => [ '', 'padding: 0.4em' ],
],
]
);
$_partnership_right = theme(
'nc_custom_templates_component_col_1',
[
'h' => [
'spacer' => '',
// '' no spacer, top or bottom
'content' => "<h3 style='${_css_h3}'><a href='${_dfp_txt_1['href']}' target='_blank' style='${_css_h3_a}'>${_dfp_txt_1['title']}</a></h3>${_dfp_txt_1['content']}",
'align' => [ '', 'left' ],
// first one is table then td
'valign' => [ '', 'top' ],
'class' => [ '', '' ],
'style' => [ '', 'padding: 0.4em' ],
],
]
);
print theme(
'nc_custom_templates_component_col_2',
[
'h' => [
'table' => [
'align' => '',
'valign' => '',
'class' => '',
'style' => '',
],
'spacer' => '',
'fixed' => FALSE,
'content' => [
$_partnership_left,
$_partnership_right
],
'align' => [ 'center', 'left' ],
'valign' => [ 'top', 'top' ],
'class' => [
'partnershipBlockContentLeft',
'partnershipBlockContentRight'
],
'style' => [ '', '' ], // padding: 0.4em
],
]
);
7.10.8. Hybrid Responsive Design
Orignal responsive design: parent table width:100%, child table.responsive-table width:600 and use media queries
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr>
<td bgcolor="#00a9f7" align="center">
<table border="0" cellpadding="0" cellspacing="0" width="600" class="responsive-table">
<tr>
<td align="center" valign="top" style="padding: 40px 0px 40px 0px;">
<!-- IMAGE -->
<img alt="Example" src="http://placehold.it/600x300" width="600" style="display: block;" border="0" class="responsive-image">
</td>
</tr>
<tr>
<td align="center" valign="top" style="padding: 0px 10px 20px 10px;">
<!-- HEADLINE -->
<p style="color: #ffffff; font-family: sans-serif; font-size: 24px; font-weight: bold; line-height: 28px; margin: 0;">Announcing Some News</p>
</td>
</tr>
<tr>
<td align="center" valign="top" style="padding: 0px 10px 60px 10px;">
<!-- COPY -->
<p style="color: #b5e2f7; font-family: sans-serif; font-size: 16px; font-weight: normal; line-height: 24px; margin: 0;">Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim.</p>
</td>
</tr>
</table>
</td>
</tr>
</table>
@media screen and (max-width: 600px) {
.responsive-table {
display: block;
width: 100% !important;
}
.responsive-image {
height: auto;
max-width: 100% !important;
}
}
Hybrid or spongy design: parent table width:100%, child table width:100% with max-width:600px, for IE, wrap another table width:600 around the child table.
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr>
<td bgcolor="#00a9f7" align="center">
<!--[if (gte mso 9)|(IE)]>
<table align="center" border="0" cellspacing="0" cellpadding="0" width="600">
<tr>
<td align="center" valign="top" width="600">
<![endif]-->
<table border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width: 600px;" >
<tr>
<td align="center" valign="top" style="padding: 40px 0px 40px 0px;">
<!-- IMAGE -->
<img alt="Example" src="http://placehold.it/600x300" width="600" style="display: block; width: 100%; max-width: 100%;" border="0">
</td>
</tr>
<tr>
<td align="center" valign="top" style="padding: 0px 10px 20px 10px;">
<!-- HEADLINE -->
<p style="color: #ffffff; font-family: sans-serif; font-size: 24px; font-weight: bold; line-height: 28px; margin: 0;">Announcing Some News</p>
</td>
</tr>
<tr>
<td align="center" valign="top" style="padding: 0px 10px 60px 10px;">
<!-- COPY -->
<p style="color: #b5e2f7; font-family: sans-serif; font-size: 16px; font-weight: normal; line-height: 24px; margin: 0;">Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim.</p>
</td>
</tr>
</table>
<!--[if (gte mso 9)|(IE)]>
</td>
</tr>
</table>
<![endif]-->
</td>
</tr>
</table>
Fab Four Technique to create responsive emails without media queries
- If width is greater than max-width, max-width wins
- If min-width is greater than width or max-width, min-width wins.
- Demo
So the following sets the box to 480px
.box {
width:320px;
min-width:480px;
max-width:160px;
}
Use calc() to set width. The following creates 2 columns that will stack and grow below 480px
.block {
display:inline-block;
min-width:50%;
max-width:100%;
width:calc((480px — 100%) * 480);
}
calc() is only not supported in latest Outlook for Windows, OWA both Office 365 and Outlook.com, and Yahoo webmail and iOS/Android apps. In these cases
.block {
display:inline-block;
min-width:240px;
width:50%;
max-width:100%;
min-width:calc(50%);
width:calc(480px * 480 — 100% * 480);
}
Final version with fixes for OWA and WebKit Prefixes
.block {
display:inline-block;
min-width:240px;
width:50%;
max-width:100%;
min-width:-webkit-calc(50%);
min-width:calc(50%);
width:-webkit-calc(230400px — 48000%);
width:calc(230400px — 48000%);
}
7.10.9. Inbox Preview
- Litmus
- covers the most email clients and used by MailChimp and Campaign Monitor. Plus plan has Live Edit and Live Preivew
- Email on Acid
- Basic plan covers unlimited tests. Fewer Android devices are supported compared with Litmus
- Testi@
- less email clients but much cheaper. Has Chrome Extension
7.10.10. Gmail Image Caching
Images are downloaded at proxy server and serve those images from that proxy instead. One download only. Affects stats of opens and geolocation. Gmail Web and Gmail iOS and Android apps.
7.10.11. CSS and HTML Support across Devices email:css:support
- https://www.campaignmonitor.com/css/
- https://templates.mailchimp.com/resources/email-client-css-support/
- https://www.caniemail.com
- Gmail Web doesn't support
<style>in<body>but supports<style>in<head> - Gmail removes support for CSS attribute selectors
- not supported in
- Browsers
- Outlook.com, Yahoo!, Gmail, AOL
<style>in<head>is not supported in:- Gmail Android app IMAP, Gmail mobile webmail, Yahoo! Mail Android app
- AOL Mail
- not supported in
- Desktop
- Outlook 2007-16, Windows 10 Mail
- Mobile
- Gmail Android app IMAP, Gmail mobile wemail but supports in Gmail Android app, Gmail iOS app
- Webmail
- AOL Mail, Outlook.com
- Don't use
andor! - some email clients don't support them e.g. Yahoo! Mail
- not supported in
- Mobile
- Gmail Android app IMAP, Gmail mobile webmail
- only supported in
- Desktop
- Outlook 2000-03, Apply Mail 10, Outlook for Mac
- Mobile
- Android Mail 4.4.4, iOS 10 Mail, iOS 11 Mail, Outlook Android app, Sparrow
- Webmail
- AOL Mail
- not supported in
- Outlook 2007-16
- Outlook Express
- Windows 10 Mail
- Windows Live Mail
- Android 4.2.2 Mail
- Gmail Android app IMAP
- Windows Phone 8 Mail
- Yahoo! Mail Android app
- Email client safe fonts
- Courier
- Courier New
- Georgia
- Times
- Times New Roman
- Refer to css:font-face:web safe fonts for checking font coverage and font stack
Target iPhone 5 and above mjml:tag:mj-hero:sample1
@media all and (max-width: 414px) { table[style*="color:#123456;"] { width:100%;} td[style*="font-family:Khand, Helvetica, sans-serif, Arial;font-size:18px;line-height:28px;color:#666465;"] { color: #000000 !important; background-color: rgba(255,255,255,0.5) !important; } tr.spacer-above-button { height:50px; } }
@media screen and (max-device-width: 320px) and (max-device-height: 568px) { /* iPhone 8 (Zoom View) */ } @media screen and (max-device-width: 375px) and (max-device-height: 667px) { /* iPhone 8 (Standard View) and iPhone 8 Plus (Zoom View) */ } @media screen and (max-device-width: 414px) and (max-device-height: 776px) { /* iPhone 8 Plus (Standard View) */ }
Prevent auto-scaling on iOS 11
<meta name="x-apple-disable-message-reformatting">
Target Samsung Mail
#MessageViewBody .yourclassnamehere { }
7.10.12. MJML
7.10.12.1. Install & Basics
npm install mjml # you can delete node_modules folder and reinstall to get the latest version ./node_modules/.bin/mjml -V # To avoid typing ./node_modules/bin/ export PATH="$PATH:./node_modules/.bin" # mjml version (e.g. 3.3.5) mjml -V # one time output html mjml input.mjml # one time output to a specific file mjml input.mjml -o my-email.html # watch a file mjml -w input.mjml # convert v3 .mjml to v4 .mjml mjml -m index.mjml indexv4.mjml # watch multiple # html files will be in views/**/*.html mjml -w source/**/*.mjml -o views # minify html output # Don't use as it will break Outlook previews node_modules/.bin/mjml path/to/index.mjml -o path/to/index.html --config.minify
- Since v4, install Windows App https://mjmlio.github.io/mjml-app/ to get WYSIWYG experience
- Install Litmus Chrome Extension and follow the instructions to tweak the extension setting. Login to Litmus.com and open a local .html file
- Install Atom packages to edit .mjml in Atom: linter-mjml and mjml-preview
- Minify html as some email clients e.g. Gmail App on iOS and Android clips the first 102kb HTML to convert and display
- Tags or mj-elements styling
- a limited set of attributes can be used
- For now, must repeat for each tag the style you want to apply on it
- Another way is to create custom component, sub-classing a standard one and have your styles defined by default
7.10.12.2. mjml mjml:tag:mjml
- Contains mjml:tag:mj-head and mjml:tag:mj-body
- Attributes
- owa
- Default none, display the mobile version for Outlook.com. To force the desktop layout on Outlook.com
<mjml owa="desktop"> - lang
- Default none. Set
<html lang="">
7.10.12.3. mj-head mjml:tag:mj-head
Contains styles and meta elements. To insert custom head elements, registerMJHeadElement(<string> name, <funciton> handler) api from mjml-core
7.10.12.4. mj-head > mj-preview mjml:tag:mj-preview
no other attributes
<mj-head>
<mj-preview>Hello MJML</mj-preview>
</mj-head>
7.10.12.5. mj-head > mj-attributes mjml:tag:mj-attributes
mj-allsets default attributes for every component inside your MJML document- inline attributes > classes > default mj-attributes > defaultMJMLDefinition
<mjml> <mj-head> <mj-attributes> <mj-text padding="0" /> <mj-class name="blue" color="blue" /> <mj-class name="big" font-size="20px" /> <mj-all font-family="Arial" /> <mj-all padding="0px"></mj-all> </mj-attributes> </mj-head> <mj-body> <mj-container> <mj-section> <mj-column> <mj-text mj-class="blue big"> Hello World! </mj-text> </mj-column> </mj-section> </mj-container> </mj-body> </mjml>
7.10.12.6. mj-head > mj-breakpoint
On which breakpoint the layout should go desktop/mobile. Default is 480px
<mjml> <mj-head> <mj-breakpoint width="320px" /> </mj-head> <mj-body> <mj-section> <mj-column> <mj-text> Hello World! </mj-text> </mj-column> </mj-section> </mj-body> </mjml>
7.10.12.7. mj-head > mj-style mjml:tag:mj-style
- Refer to email:css:support
mj-styleinline="inline"When
inline="inline"used, don't use double quotes insidemj-style<mjml> <mj-head> <mj-style> @media (max-width: 480px) { div[style*="color:#F45e46;"] { text-align: center !important } } </mj-style> <mj-style inline="inline"> .link-nostyle { color: inherit; text-decoration: none } <!-- for a tag, color: inherit is not working, either specify color in .link-nostyle or style="color:#fff" in a tag. --> </mj-style> </mj-head> <mj-body> <mj-container> <mj-section> <mj-column> <mj-image width="100" src="/assets/img/logo-small.png"></mj-image> <mj-divider border-color="#F45E43"></mj-divider> <mj-text font-size="20px" color="#F45e46" font-family="helvetica"> Hello <a href="https://mjml.io" class="link-nostyle">World</a> </mj-text> </mj-column> </mj-section> </mj-container> </mj-body> </mjml>
mjml:tag:mj-style:responsive font size
<mjml> <mj-head> <mj-style> .body-fix div { font-size: 22px !important; /* This will override font-size attribute if client supports style in head.*/ /* Define styles use mjml elements' attribute for devices don't support <style> in <head>: Gmail Android app IMAP, Gmail mobile webmail, Yahoo! Mail Android app, Google Inbox, Android 5.1+ Native Mail App */ line-height: 28px !important; } @media only screen and (max-width: 480px){ .body-fix div { font-size: 28px !important; line-height: 44px !important; } } </mj-style> </mj-head> <mj-body> <mj-container> <mj-section> <mj-column> <mj-text css-class="body-fix" font-size="20px" color="#F45E43" font-family="helvetica">Will work :)</mj-text> </mj-column> </mj-section> </mj-container> </mj-body> </mjml>
Use
css-classin anymj-elementand change styles for mobile e.g. hide on mobile<mj-style> @media all and (max-width:480px) { tr[class*=hidden] { display:none; } <!-- Better --> *[class*=hidden] { display:none !important;} } <!-- Hide column on mobile: - Desktop clients: all ok - Mobile: only Gmail IMAP app displays the column while it should not show - Web browser: all ok --> .hide-on-mobile {display:none !important;} @media (min-width:480px) { .hide-on-mobile { display: inline-block !important;} } <!-- Hide column on mobile: My Version as the above solution is hard to comprehend - Desktop clients: all ok - Mobile: only Gmail IMAP app displays the column while it should not show - Web browser: all ok --> .hide-on-mobile {display:none !important;} .hide-on-mobile-outlook .hide-on-mobile {display: inline-block !important;} @media (min-width:480px) { .hide-on-mobile {display:inline-block !important;} .hide-on-mobile-outlook .hide-on-mobile {display: inline-block !important;} } </mj-style> <!-- Show on mobile only, hide on desktop - Desktop clients: all hidden (good!) - Mobile clients: only Gmail App IMAP does not show, all mobile clients show (good!) - Web browser: all hidden (Good!) --> <mj-style inline="inline"> .show-on-mobile-only, .show-on-mobile-only-outlook {display:none;} .show-on-mobile-only table, .show-on-mobile-only-outlook table {mso-hide:all;} </mj-style> <mj-style> @media (max-width:480px) { .show-on-mobile-only, .show-on-mobile-only-outlook { display: inline-block !important;} } </mj-style> <!-- Show on mobile only, hide on desktop. --> <mj-text css-class="hidden"></mj-text> <mj-column css-class="hide-on-mobile"></mj-column> <mj-section css-class="show-on-mobile-only"></mj-section>
css-class only works for
mj-*elements, use this to target elements inside mj-text<mj-style> @media all and (max-width: 361px) { h2[class*="header"] { font-size:50px!important;} } @media all and (max-width: 321px) { h2[class*="header"] { font-size:49px!important;} } </mj-style> <!-- h2[class*="header"] { font-size:55px!important;} --> <mj-text css-class="main-content" color="#001f58" padding="0px 0px 0px 0px"> <h2 css-class="main-content" class="header" style="font-size:55px;line-height:50px;font-family:'Trebuchet MS',sans-serif;padding:0;margin:0;font-weight:100;padding:0;margin:0;">...</h2> </mj-text>
7.10.12.8. mj-head > mj-font mjml:tag:mj-font
<mjml> <mj-head> <mj-font name="Raleway" href="https://fonts.googleapis.com/css?family=Raleway" /> </mj-head> <mj-body> <mj-container> <mj-section> <mj-column> <mj-text font-family="Raleway, Arial"> Hello World! </mj-text> </mj-column> </mj-section> </mj-container> </mj-body> </mjml>
7.10.12.9. mj-head > mj-title mjml:tag:mj-title
<mj-head> <mj-title>Hello MJML</mj-title> </mj-head>
7.10.12.10. mj-head > mj-body > mj-section (mj-container)
mjml:tag:mj-body mjml:tag:mj-section mjml:tag:mj-container
- To insert custom elements either registerMJElement(<MJMLElement> class) api from mjml-core or via .mjmlconfig file
- Non-known element from mjml-core are ignored
- can have multiple
mj-section. For v3,mj-bodyshould have only one root element which usually ismj-container, which contains the entire content- Attributes
- width
- default 600px
- background-color
- default n/a. Don't set to white as email clients may use Dark Mode
- (no term)
- css-class
- Attributes
- There's no more
mj-containerin version 4.mj-sectionwasmj-containerin version 3 andmj-containerwasmj-bodyin version 1.x- Attributes
- full-width
- default n/a. If set to full-width then section is extended to the full width beyond 600px
- (no term)
- background-color
- (no term)
- background-url
- background-size
- auto
- background-repeat
- repeat
- vertical-align
- top
- text-align
- center
- padding
- 20px 0
- direction
- ltr (or rtl)
- rendered HTML is class="xx"
- Attributes
7.10.12.11. mj-body > mj-include mjml:tag:mj-include
Inside mjml > mj-body > mj-container <mj-include path="./header" /> <!– or 'header.mjml' –>
7.10.12.12. mj-body > mj-section mjml:tag:mj-section
- Should contain mj-column. Up to 4 columns
- Section has up to 600px width
- May use % for left and right padding
- Structure the columns in the order you want them to stack on mobile, then use direction attribute to change the order on desktop
- Refer to https://mjml.io/documentation/#mjml-section
- full-width
- n/a (full-width)
- direction
- ltr or rtl, default ltr
- padding
- default 20px 0
- (no term)
- padding-top, padding-x
- background-url
- default n/a
- background-repeat
- default repeat, no-repeat
- background-color
- default n/a,
7.10.12.13. mj-wrapper > mj-section mjml:tag:mj-wrapper
- Wrap multiple sections. Perfect place to set borders
- Refer to https://mjml.io/documentation/#mjml-wrapper
- padding
- 20px 0
- (no term)
padding-*- background-url
- If mj-wrapper has this defined, don't define in its mj-section
- background-repeat
- repeat
- background-size
- auto
- border
- none
- direction
- ltr (rtl)
- vertical-align
- top
- text-align
- center
- full-width
- Don't add full-width="full-width" inside mj-section as it will make mj-section non-full-width
- I also find out if there are other sections that are also inside mj-container, define full-width only for that wrapper and don't define it for other sections
- Sometimes mj-section does not have to be full-width, you can set left/right padding to 0 for mj-section
- (no term)
- css-class
<mjml> <mj-body> <mj-section></mj-section> <mj-wrapper border="1px solid #000000" padding="50px 30px"> <mj-section border-top="1px solid #aaaaaa" border-left="1px solid #aaaaaa" border-right="1px solid #aaaaaa" padding="20px"> <mj-column> <mj-image padding="0" src="https://placeholdit.imgix.net/~text?&w=350&h=150" /> </mj-column> </mj-section> <mj-section border-left="1px solid #aaaaaa" border-right="1px solid #aaaaaa" padding="20px" border-bottom="1px solid #aaaaaa"> <mj-column border="1px solid #dddddd"> <mj-text padding="20px"> First line of text </mj-text> <mj-divider border-width="1px" border-style="dashed" border-color="lightgrey" padding="0 20px" /> <mj-text padding="20px"> Second line of text </mj-text> </mj-column> </mj-section> </mj-wrapper> </mj-body> </mjml>
- mj-container > mjml:tag:mj-column
Any mj-element included in a column will have a width equals to 100% of this column's width. e.g. mj-text
Inside a column, any standard element, or the ones you've defined and registered can be included.
But don't include another column or section.
- background-color
- default n/a
- Auto sizing
2 columns, each column 50%, 3 columns each is 33%, etc.
- Manual sizing
<mj-column width="300px"> <!-- First column content --> </mj-column> <mj-column> <!-- Second column content --> </mj-column>
Once one column's width is set, you have to set manually each column size.
Always keep in mind that the maximum space to end up with a responsive layout is 600px.
- vertical-align
- default top, can be middle/bottom
- width
- default (100/number of columns %)
No padding is set by default
7.10.12.14. mj-container > mj-hero mjml:tag:mj-hero
https://mjml.io/documentation/#mjml-hero
<mj-container> <mj-hero mode="fluid-height" background-width="600px" background-height="469px" background-url="https://via.placeholder.com/600x469/?text=600x469" background-color="#2a3448" padding="100px 0px"> <!-- To add content like mj-image, mj-text, mj-button ... use the mj-hero-content component --> <mj-hero-content width="100%"> <mj-text padding="20px" color="#ffffff" font-family="Helvetica" align="left" font-size="45" line-height="45px" font-weight="900"> GO TO SPACE </mj-text> <mj-button href="https://mjml.io/" align="left"> ORDER YOUR TICKET NOW </mj-button> </mj-hero-content> </mj-hero> </mj-container>
Always specify background-color mode :: fluid-height or fixed-height
<mj-container> <mj-hero mode="fixed-height" height="469px" background-width="600px" background-height="469px" background-url="https://via.placeholder.com/600x469/?text/600x469" background-color="#2a3448" padding="100px 0px"> <!-- To add content like mj-image, mj-text, mj-button ... use the mj-hero-content component --> <mj-hero-content width="100%"> <mj-text padding="20px" color="#ffffff" font-family="Helvetica" align="center" font-size="45" line-height="45px" font-weight="900"> GO TO SPACE </mj-text> <mj-button href="https://mjml.io/" align="center"> ORDER YOUR TICKET NOW </mj-button> </mj-hero-content> </mj-hero> </mj-container>
mj-hero can be a mj-section mjml:tag:mj-hero:sample1
<mj-hero mode="fluid-height" height="420px" background-width="600px" background-height="420px" background-url="http://600x420.jpg" background-color="#FFFFFF" padding="0px"> <mj-hero-content width="100%"> <mj-text font-size="18px" font-family="Khand, Helvetica, sans-serif, Arial" font-weight="900"> Title aligned to left </mj-text> <mj-table width="55%" color="#123456"> <tr style="list-style: none;line-height:1"> <td style="font-family:Khand, Helvetica, sans-serif, Arial;font-size:18px;line-height:28px;color:#666465;"> This text aligned to the left and takes 55% width </td> </tr> </mj-table> <mj-spacer height="20px" css-class="spacer-above-button" /> <mj-button background-color="#FF0025" font-size="16px" align="left" font-family="'Merriweather Sans', Helvetica, sans-serif, Arial" padding-left="25px"> Learn more </mj-button> <mj-spacer height="20px" /> <mj-text font-size="14px" font-weight="900" font-family="'Merriweather Sans', sans-serif;"> Phone number <a href="http://a.com" style="color:#000000;text-decoration:none;">a.com</a> </mj-text> <mj-social color="#333333" align="left" text-mode="false" icon-size="16px" padding-left="25" padding-bottom="10" padding-top="10" mode="horizontal" display="linkedin:url facebook:url twitter:url youtubec:url" linkedin-href="https://www.linkedin.com/" linkedin-content=" " linkedin-icon-color="#A7A6A4" facebook-href="https://www.facebook.com/" facebook-content=" " facebook-icon-color="#A7A6A4" twitter-href="https://twitter.com/" twitter-content=" " twitter-icon-color="#A7A6A4" youtubec-href="https://www.youtube.com/" youtubec-content=" " youtubec-icon-color="#A7A6A4" youtubec-icon="http://16x16.jpg" /> <mj-spacer height="30px" /> </mj-hero-content> </mj-hero> <mj-section> ... </mj-section>
7.10.12.15. mj-section > mj-column mjml:tag:mj-column
- Attributes
- width
- without it,
mj-columnis evenly distributed. Can be pixels or percentage - vertical-align
- top
- padding,
padding-* - 20px 0
- css-class
- border,
border-* - backgrond-color
7.10.12.16. mj-elements inside mj-column
- Any
mj-elementinsidemj-columnwill have width equivalent to 100% of this column's width - Any
mj-elementor elements insidemj-rawincludingmj-imageshould not have a higher width than the current column, otherwise Outlook will have troubhle in laying outmj-column
- mj-text mjml:tag:mj-text
- It can contain any HTML tag with any attributes
- 10px 25px
- #000000
- n/a
- Ubuntu, Helvetica, Arial, sans-serif
- 13px
- font-style
- 22px. Use %.
- none
- left
Basic
<mj-section background-color="#f0f0f0"> <mj-column> <mj-text font-style="italic" font-size="20" color="#626262"> My Company <a href="#" style="text-decoration:none!important;text-decoration:none;color:#2DDCB4">... read more</a> </mj-text> <mj-text> <h1>Hello Title</h1> </mj-text> </mj-column> </mj-section>
- mjml:tag:mj-style:responsive font size
- https://litmus.com/blog/update-banning-blue-links-on-ios-devices
<meta content="telephone=no" name="format-detection">6‌75 Massachusetts Ave, Cambridge, MA 02139is675 Massachusetts Ave, Cambridge, MA 02139a[href^=tel]{ color:#F00; text-decoration:none;}
- Change styles for all iOS data detection!
- iOS add an attribute to all links
<a href="#" x-apple-data-detectors="true">- (no term)
So add style
a[x-apple-data-detectors] { color: inherit !important; text-decoration: none !important; /* below is optional */ font-size: inherit !important; font-family: inherit !important; font-weight: inherit !important; line-height: inherit !important; } // Samsung Mail automatically adds id MessageViewBody to <body> and a <div id="MessageWebViewDiv"> #MessageViewBody a { color: inherit; text-decoration: none; font-size: inherit; font-family: inherit; font-weight: inherit; line-height: inherit; }
- Customize line-height for Outlook desktop clients only
- Define ling-height in mj-text attribute and then set @media
- line-height for Outlook desktop clients is 95% while others support @media use 83%
This can be used to target other CSS property for Outlook desktop clients only
<mj-style> @media all and (min-width: 500px) { .main-content-h1-lh div { line-height:83%!important; } } </mj-style> <mj-text css-class="main-content-h1-lh" line-height="95%">...</mj-text>
- mj-button, Image header with text and button
- href
- link has to be valid!
- target
- _blank
- padding
- 10px 25px
- inner-padding
- 10px 25px
- background-color
- #414141
- container-background-color
- n/a
- color
- #ffffff
- border
- none
- border-radius
- 3px
- font-size
- 13px
- font-weight
- normal
- font-family
- Ubuntu, Helvetic, Arial, sans-serif
- line-height
- 120%, px/%/none
- align
- center
- vertical-align
- middle
- text-align
- none
- text-decoration
- none, underline/overline/none
- text-transform
- none, capitalize/uppercase/lowercase
- rel
- link rel attribute
- (no term)
- css-class
<mj-section background-url="http://1.bp.blogspot.com/-TPrfhxbYpDY/Uh3Refzk02I/AAAAAAAALw8/5sUJ0UUGYuw/s1600/New+York+in+The+1960's+-+70's+(2).jpg" background-size="cover" background-repeat="no-repeat"> <!-- -In order to have the background rendered full-width in the column, set the column width to 600px--> <mj-column width="600"> <mj-text align="center" color="#fff" font-size="40" font-family="Helvetica Neue">Slogan here</mj-text> <mj-button background-color="#F63A4D" href="#"> Promotion </mj-button> </mj-column> </mj-section>
- mj-image mjml:tag:mj-image
- Attributes
- width
px, default 100% (100% of the column width)
mj-imagewidth is is 300px,mj-imageleft/right total padding is 0px, column width ofmj-imageis690*0.5=345px
- On desktop, image display width is
min[ Column_Width - MJ_Image_Paddings, MJ_Image_Width ] = min[ 345-0=345, 300 ] = 300px - On mobile, image display width is
min[ 100% vw - 50, Column_Width - MJ_Image_Paddings, MJ_Image_Width ] = 300x works on all clients except Yahoo! Mail on web
<mj-style inline="inline"> .img-300x169 {width:300px;height:169px;} </mj-style> <mj-image css-class="img-300x169" width="300px" height="169px" src="..." alt="" />
- height
- auto
- align
- center
- padding
- 10px 25px
- (no term)
padding-*- border
- none
- border-width
- default 4px
- border-style
- default solid, dashed/dotted
- border-color
- default #000000
- alt
- string
- (no term)
- href
- target
- _blank
- srcset
- only supported in iOS. Determine display size and multiply the dimension by 2
When mj-image is inside a mj-column that is the only column, then size doesn't matter. If mj-image is inside a column that has other sibling columns, then limit width so that columns are layed out correctly in Outlook.
<mj-section background-color="white"> <!-- Left image --> <!-- column is 50% of total width, mj-image is vertically and horizontally centered and image width is always 200 --> <mj-column> <mj-image width="200px" src="http://via.placeholder.com/300x400/?text=300x400" /> </mj-column> <!-- right paragraph --> <mj-column> <mj-text font-style="italic" font-size="20" font-family="Helvetica Neue" color="#626262"> Find amazing places </mj-text> <mj-text color="#525252"> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin rutrum enim eget magna efficitur, eu semper augue semper. Aliquam erat volutpat. Cras id dui lectus. Vestibulum sed finibus lectus.</mj-text> </mj-column> </mj-section>
- Attributes
- mj-social mjml:tag:mj-social
- Refer to mjml:tag:mj-table:social
Version 4
<mj-section background-color="#e7e7e7"> <mj-column> <mj-social mode="horizontal"> <mj-social-element name="facebook" /> <mj-social-element name="twitter-noshare" href="..." src="..." alt="" background-color="#3d3d3d" /> </mj-social> </mj-column> </mj-section>
mj-social- attributes
- align
- center
- border-radius
- 3px
- font-family
- Ubuntu, Helvetica, Arial, sans-serif
- font-size
- 13px
- icon-size
- 20px. percent/px.
- icon-height
- 20px. percent/px. Overrides icon-size
- icon-padding
- 0px
- line-height
- 22px
- inner-padding
- 4px
- padding
- 10px 25px
- container-background-color
- n/a
- mode
- horizontal (vertical)
- text-decoration
- none
- text-padding
- 4px 4px 4px 0
- color
- #333333
Version 3
<mj-social color="#333333" align="left" text-mode="false" icon-size="16px" padding-right="25" padding-bottom="10" padding-top="10" mode="horizontal" display="linkedin:url facebook:url twitter:url youtubec:url" linkedin-href="..." linkedin-content=" " linkedin-icon-color="#A7A6A4" youtubec-href="..." youtubec-content=" " youtubec-icon-color="#A7A6A4" youtubec-icon="path to youtube icon image" />
mj-socialattributes
- text-mode
- default true, if true, *-content must be defined
- (no term)
- font-family
- color
- text color, default #333333
- align
- center
- icon-size
- default 20px
- padding
- default 10px 25px
- mode
- defualt horizontal, or vertical
- display
- default facebook twitter google. could be facebook:url, can add custom social network e.g. youtube
*-href,*-content,*-rel,*-icon,*-icon-color*could be facebook, twitter, google, linkedin, pinterest, instagram
*-iconfor default facebook, twitter, etc. cannot be modified*-icon-colorhas to be defined for custom social medias e.g. youtubec
- mj-divider
Try to add horizontal border with some spacing using mj-divider
- border-color
- #000000
- border-style
- solid, dashed, dotted
- border-width
- 4px
- width
- 100% px/percent
- padding
- 10px 25px
- container-background-color
<mj-section padding-bottom="0" background-color="white"> <mj-column width="100%"> <mj-image src="https://avatars0.githubusercontent.com/u/16115896?v=3&s=200" width="50px" /> <mj-divider padding-top="20" padding-bottom="0" padding-left="0" padding-right="0" border-width="1px" border-color="#f8f8f8" /> </mj-column> </mj-section>
- mj-spacer
blank space. For some reason, try adding a section with a spacer only so that the email has the content in full width
- height
- 20px
- (no term)
- css-class
<mj-section> <mj-column> <mj-text>A first line of text</mj-text> <mj-spacer height="50px" /> <mj-text>A second line of text</mj-text> </mj-column> </mj-section>
- mj-table
- only accepts plain html
- Use
mj-tableto have fixed width that's not gonna change when in mobile. Or width will be 100% mj-tablealways, as a whole, align left while mjml:tag:mj-social aligns center. To center, usemj-rawto mjml:raw:center table- Attributes
- font-family
- Ubuntu, Helvetica, Arial, sans-serif
- font-size
- 13px
- line-height
- 22px (%/px)
- padding
- 10px 25px
- width
- default 100%. With default left+right padding 25*2, the maximum width can be less than 300px. Otherwise the table has 100% width
- table-layout
- auto (fixed/initial/inherit)
mjml:tag:mj-table:social Use it to align icons with links. Notice the img has fixed width.
mj-tablehas fixed width including padding 25*5+5*(5-1)=165px. Ifmj-tablehas no width, columns will have equal width.<mj-table width="165px"> <!-- the style in tr is very important for Outlook to have the correct column height--> <tr style="list-style: none;line-height:1"> <td> <a href="https://twitter.com/RecastAI"> <img width="25" src="https://cdn.recast.ai/newsletter/twitter.png" /> </a> </td> <td> <a href="https://www.facebook.com/recastAI"> <img width="25" src="https://cdn.recast.ai/newsletter/facebook.png" /> </a> </td> <td> <a href="https://medium.com/@RecastAI"> <img width="25" src="https://cdn.recast.ai/newsletter/medium.png" /> </a> </td> <td> <a href="https://www.youtube.com/channel/UCA0UZlR8crpgwFiVaSTbVWw"> <img width="25" src="https://cdn.recast.ai/newsletter/youtube.png" /> </a> </td> <td> <a href="https://plus.google.com/u/0/+RecastAi"> <img width="25" src="https://cdn.recast.ai/newsletter/google%2B.png" /> </a> </td> </tr> </mj-table>
- For borders on 4 sides (bottom/top/left/right), I had trouble with setting in
<tr>. Define 4 sided borders in<td>is fine.
<mj-column> <mj-table> <tr style="border-bottom:1px solid #ecedee;text-align:left;padding:15px 0;"> <th style="padding: 0 15px 0 0;">Year</th> <th style="padding: 0 15px;">Language</th> <th style="padding: 0 0 0 15px;">Inspired from</th> </tr> <tr> <td style="padding: 0 15px 0 0;">1995</td> <td style="padding: 0 15px;">PHP</td> <td style="padding: 0 0 0 15px;">C, Shell Unix</td> </tr> <tr> <td style="padding: 0 15px 0 0;">1995</td> <td style="padding: 0 15px;">JavaScript</td> <td style="padding: 0 0 0 15px;">Scheme, Self</td> </tr> </mj-table> </mj-column>
<mj-section> <mj-column> <mj-table> <tr style="list-style: none;line-height:1"> <td width="46"><img width="46" src="http://via.placeholder.com/46x32/" /></td> <td width="10"> </td> <td> <br> <span style="font-size:18px;font-weight:200;">Contact Us</span><br> <a href="tel:18881231234" target="_blank" style="text-decoration:none;color:#545759;">1-888-123-1234</a><br> <a href="mailto:a@xyz.com" target="_blank" style="text-decoration:none;color:#545759;">a@xyz.com</a><br> <a href="http://a.com" target="_blank" style="text-decoration:none;color:#545759;">www.a.com</a> </td> </tr> </mj-table> </mj-column> </mj-section>
- mj-raw
- Not to be rendered by MJML engine
- Unclosed HTML tags inside
mj-rawwill be closed - No attributes
mjml:raw:center table This won't work for Yahoo! Mail but works on all clients
<mj-column> <mj-text>...</mj-text> <!-- each mj-element under mj-column is <tr><td> content </td></tr> --> <!-- align center --> <mj-raw> <tr> <td> <table width="100%" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top"> <tr> <td align="center" style="font-size:0px;padding:10px 25px;padding-bottom:0px;word-break:break-word;"> <!-- social icons --> <table width="95" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top"> <tr style="list-style:none;line-height:1;"> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> </tr> </table> <!-- social icons. --> </td> </tr> </table> </td> </tr> </mj-raw> <!-- align center. --> <!-- align right, pay attention to padding-right 25px --> <mj-raw> <tr> <td> <table width="100%" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top"> <tr> <td align="right" style="font-size:0px;padding:10px 25px;padding-bottom:0px;word-break:break-word;"> <!-- social icons --> <table width="95" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top"> <tr style="list-style:none;line-height:1;"> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> <td> <a href="#" target="_blank"> <img width="20" height="20" src="#" alt="" /> </a> </td> </tr> </table> <!-- social icons. --> <!-- mj-image with href HTML rendered result --> <table cellpadding="0" cellspacing="0" border="0" role="presentation" style="border-collapse:collapse;border-spacing:0px;"> <tr> <td style="width:110px;"> <a href="#" target="_blank"> <img width="110" height="auto" src="#" alt="" style="border:0;display:block;outline:none;text-decoration:none;height:auto;width:100%;" /> </a> </td> </tr> </table> <!-- mj-image with href HTML rendered result. --> </td> </tr> </table> </td> </tr> </mj-raw> <!-- align right, pay attention to padding-right 25px. --> </mj-column>
- Old align center
<mj-style inline="inline"> .tbl-cell-1 { font-family:'Helvetica Neue', Helvetica, Arial, sans-serif; font-size:12px; padding:0px 0px 10px 0px; width:30px; } .tbl-cell-2, .tbl-cell-3 { width:30px; } </mj-style> <mj-column> <mj-text>...</mj-text> <!-- each mj-element under mj-column is <tr><td> content </td></tr> --> <!-- OLD align center --> <mj-raw> <tr> <td> <table align="center" width="110" cellpadding="0" cellspacing="0" border="0"> <tr> <td align="center" class="tbl-cell-1"><a href="#"> <img width="30" src="twitter.png" /> </a></td> <td align="center" class="tbl-cell-2"><a href="#"> <img width="30" src="twitter.png" /> </a></td> <td align="center" class="tbl-cell-3"><a href="#"> <img width="30" src="twitter.png" /> </a></td> </tr> </table> </td> </tr> </mj-raw> <!-- OLD align center. --> </mj-column>
- mj-hero
Display a section with a background image and some content inside (mj-text, mj-button, mj-image …) wrapped in mj-hero-content component
- mj-invoice
- mj-location
- mj-navbar
https://mjml.io/documentation/#mjml-navbar So far, all have been tested only show expanded items.
- mj-carousel
https://mjml.io/documentation/#mjml-carousel Outlook and OWA only shows the first image. Works for Gmail.com and Android Gmail.
- mj-accordion
https://mjml.io/documentation/#mjml-accordion Every attribute in `mj-accordion` are applied to `mj-accordion-element` unless you overide them on `mj-accordion-element`
So far, all have been tested only show expanded items.
The "hamburger" feature only work on mobile device with all iOS mail client, for others mail clients the render is performed on an normal way, the links are displayed inline and the hamburger is not visible.
<mjml> <mj-head> <mj-attributes> <mj-accordion border="none" padding="1px" /> <mj-accordion-element icon-wrapped-url="http://i.imgur.com/Xvw0vjq.png" icon-unwrapped-url="http://i.imgur.com/KKHenWa.png" icon-height="24px" icon-width="24px" /> <mj-accordion-title font-family="Roboto, Open Sans, Helvetica, Arial, sans-serif" background-color="#fff" color="#031017" padding="15px" font-size="18px" /> <mj-accordion-text font-family="Open Sans, Helvetica, Arial, sans-serif" background-color="#fafafa" padding="15px" color="#505050" font-size="14px" /> </mj-attributes> </mj-head> <mj-body> <mj-section padding="20px" background-color="#ffffff"> <mj-column background-color="#dededd"> <mj-accordion> <mj-accordion-element> <mj-accordion-title>Why use an accordion?</mj-accordion-title> <mj-accordion-text> <span style="line-height:20px"> Because emails with a lot of content are most of the time a very bad experience on mobile, mj-accordion comes handy when you want to deliver a lot of information in a concise way. </span> </mj-accordion-text> </mj-accordion-element> <mj-accordion-element> <mj-accordion-title>How it works</mj-accordion-title> <mj-accordion-text> <span style="line-height:20px"> Content is stacked into tabs and users can expand them at will. If responsive styles are not supported (mostly on desktop clients), tabs are then expanded and your content is readable at once. </span> </mj-accordion-text> </mj-accordion-element> </mj-accordion> </mj-column> </mj-section> </mj-body> </mjml>
7.10.12.17. mj-group to group columns
Prevent columns from stacking on mobile.
Columns inside a group must have a width in percentage or auto sizing, but not in pixel
Section can have both mj-column and mj-group.
iOS 9 :: always remove space between columns inside a mj-group. Or export HTML using minify.
- width
- 100 / number of children in section (%/px)
- vertical-align
- top
- background-color
- default n/a
7.10.12.18. Background Image
background-url can only be used in mj-hero, mj-section, mj-wrapper background-position can only be set in mj-hero and top center for other mjml elements
7.10.12.19. Change default padding
Use this to change left/right padding in % is not consistent especially in Outlook 2003-2016. For Outlook desktop, you may need to hard set mj-image width.
<mjml>
<mj-head>
<mj-attributes>
<mj-all padding-left="0px"></mj-all>
<mj-all padding-right="0px"></mj-all>
</mj-attributes>
</mj-head>
<mj-section padding-left="12%" padding-right="12%">
</mj-section>
7.10.12.20. Caveats
mj-imageinside mj-column will always have a maximum fixed width- If there's one
mj-column,mj-imageis 100% of the mj-column - If there're 2
mj-column,mj-imageis 50% of the whole width = 300px - If there're 3
mj-column,mj-imageis 196px - You can align
mj-imageleft or right inside amj-column. And when you do that, pay attention to the padding. Usually don't setpadding:0 When in mobile,
<p>haspadding-left:10px. So you could do<mj-section full-width="full-width"> <mj-group> <mj-column width="50%"> <mj-image src="logo1.jpg" width="103" align="left" padding="10px 0 0 10px" /> </mj-column> <mj-column width="50%"> <mj-image src="logo2.jpg" width="96" align="right" padding="10px 10px 0 0"/> </mj-column> </mj-group> </mj-section>
- If there's one
mj-section, mj-column don't have left and right padding by default.
mj-image has padding:10px 25px by default
7.10.13. Non breaking css:content
7.10.14. Remove spaces between HTML end tags and start tags email:spaces between html tags
// m flag for multiple lines // IE start and end if statements need a speparate line // <tr><td> // <!--[if mso | IE]><table><tr><td> // <!--<![endif]--> $mini_code = preg_replace('~>\s+<(?!\!)~m','><',$code);
7.10.15. Don't use anchor link!
7.11. Phone number
<a href="tel:+18475555555">1-847-555-5555</a>
// javascript window.open('tel:+11231231234');
7.12. Link html:link
- Also can be set in HTML response header as Link. Refer to header:link
7.12.1. media="print" html:link:media
- Refer to css:@media
<link rel="stylesheet" href="print.css" media="print">
7.12.2. rel="preload" html:link:rel:preload
- Not supported in IE11-, Edge, Safari 11-, iOS Safari 11.2-
- Unsupported browsers will just ignore
- Resources loaded via
<link rel="preload">are stored locally in the browser, and are effectively inert until they're referenced in the DOM, JavaScript, or CSS - Does not block
window.onloadevent - To load late-discovered critical resources which will be loaded very soon at current request
- Compliant with
Content-Security-Policyand http headerAccept Preload script
<link rel="preload" href="used-later.js" as="script" type="text/javascript" /> <!-- json --> <link rel="preload" href="/graphql/query?id=12345" as="fetch" type="application/json" /> <!-- ...other HTML... --> <script> // Later on, after some condition has been met, we run the preloaded // JavaScript by inserting a <script> tag into the DOM. var usedLaterScript = document.createElement('script'); usedLaterScript.src = 'used-later.js'; document.body.appendChild(usedLaterScript) </script>
<link rel="preload" as="image" href="logo.jpg"/>- html:link:rel:preload:font
<link rel="preload" href="font.woff2" as="font" type="font/woff2" crossorigin>- Font needs
crossorigineven it's on the same origin.typeis also needed - All browsers that support preloading also support WOFF2, so
font/woff2is the format that you should preload - preload won't have the ability to css:font-face:local, so the font will be always downloaded
- html:video:preload
If you don't want to preload a resource at page load but want to preload it before it's used
var preload = document.createElement("link"); link.href = "myscript.js"; link.rel = "preload"; link.as = "script"; document.head.appendChild(link); // later var script = document.createElement("script"); script.src = "myscript.js"; document.body.appendChild(script);
do something when resource is downloaded which is different from
window.onload<link rel="preload" as="style" href="async_style.css" onload="this.rel='stylesheet'"> <script> var asyncScriptOnLoad = function() { var script = document.createElement('script'); script.src = this.href; document.body.appendChild(script); }; </script> <link rel="preload" as="script" href="async_script.js" onload="asyncScriptOnLoad();">
Detect if
<link rel="preload">is supportedconst preloadSupported = () => { const link = document.createElement('link'); const relList = link.relList; if (!relList || !relList.supports) return false; return relList.supports('preload'); };
- HTTP/2 Server Push can push resources to browser before browser sends the request for them. Which means Push can send resources before HTML even started to be sent to the browser. Resources have to be on the same origin
7.12.3. rel="prefetch"
- Has better browser support than
rel="preload" - Download non-critical resources earlier
- Prefetch non-recursively fetches the top level resource not the child resources
Downloads the same css 2 times
<link rel="prefetch" href="optional.css"> <link rel="stylesheet" href="optional.css">
7.12.4. rel="dns-prefetch"
rel="preconnect"has less browser support and it's a superset of dns-prefetch<link rel="dns-prefetch" href="http://www.spreadfirefox.com/"> <link rel="dns-prefetch" href="//www.spreadfirefox.com">
7.12.5. preload vs prefetch and Network Prioritisation
- Network Prioritisation
- https://medium.com/reloading/preload-prefetch-and-priorities-in-chrome-776165961bbf
- (no term)
- Priority shown in DevTools:
- Highest
- layout-blocking
- CSS inside
<head> - match
- (no term)
- Font
- (no term)
- XHR (sync)
- (no term)
<link rel=preload as=style>
- CSS inside
- High
- load in layout-blocking phase
<script>inside<head>- requested before any non-preloaded images
- (no term)
<link rel=preload as=script>- (no term)
<link rel=preload as=font>- (no term)
- Import
- (no term)
- Image (in viewport)
- (no term)
<link rel=preload>with no as will behave as XHR- (no term)
- XSL
- (no term)
- XHR/fetch* (async)
- Medium
- Load one-at-a-time in layout-blocking phase
<script>- requested after any non-preloaded images
- (no term)
- Favicon
- Low
- Load one-at-a-time in layout-blocking phase
<script async><script defer>- Injected
<script> - Image
<link rel=preload as="image">- Media
- SVG
- Lowest
- Load one-at-a-time for layout-blocking phase. after current page is done loading and there's bandwidth available
- CSS
- mismatch
- (no term)
<link rel="prefetch">
7.12.6. rel="canonical" and rel="alternate" html:link:rel:canonical html:link:rel:alternate
- A canonical URL is the URL of the page that Google thinks is most representative from a set of duplicate pages
- Different language versions of a single page are considered duplicates only if the main content is in the same language (that is, if only the header, footer, and other non-critical text is translated, but the body remains the same, then the pages are considered to be duplicates). In general, a translated page is not a duplicate
- Use absolute paths
- 301 redirect is the best way to ensure other duplicate pages are removed and directed to the canonical URL
<!-- desktop page: http://www.example.com/page-1 --> <link rel="alternate" media="only screen and (max-width: 640px)" href="http://m.example.com/page-1"> <!-- mobile page: http://m.example.com/page-1 --> <link rel="canonical" href="http://www.example.com/page-1"> <!-- sitemap --> <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"> <url> <loc>http://www.example.com/page-1/</loc> <xhtml:link rel="alternate" media="only screen and (max-width: 640px)" href="http://m.example.com/page-1" /> </url> </urlset>
7.12.7. rel="manifest.json" html:link:rel:manifest
- Required by Crhome to show the Add to Home Screen prompt
- DevTools > Application > Manifest
- https://developers.google.com/web/fundamentals/web-app-manifest/
- icons as png
- 120
- Android
- iOS
- another iOS
- 180
- 192
- Chrome desktop
<link rel="manifest" href="/manifest.json">
{
"short_name": "Maps", // 12 characters
"name": "Google Maps", // your app name, 45 characters
// If you want to have your own size, provide icons in increments of 48px or 128px.
"icons": [
{
"src": "/images/icons-192.png",
"type": "image/png",
"sizes": "192x192"
},
{
"src": "/images/icons-512.png",
"type": "image/png",
"sizes": "512x512"
}
],
"start_url": "/?utm_source=pwa", // the page the app should start with. Tracking query string can be added e.g. track how often the app is launched.
"background_color": "#3367D6",
"display": "standalone",
// standalone: separte from the browser and runs in its own window.
// fullscreen: no browser UI
// minimal-ui: not supported by Chrome. Similar to fullscreen but might have other navigation (back, forward, reload)
// browser: a standard browser experience
// "orientation": "landscape",
// Enforce an orientation
"scope": "/maps/",
// if current page is not inside this directory, then the user exists the app
// If scope is not defined, then default is the directory that the app manifest is served from
// can be relative path e.g. ../
// start_url must be in the scope
// start _url is relative to the path defined in scope
// start_url starting with / will always be the root of the origin
"theme_color": "#3367D6"
}
7.12.8. rel="icon" rel="apple-touch-startup-image" rel="apple-touch-icon"
Refer to html:address bar https://developers.google.com/web/fundamentals/design-and-ux/browser-customization/
Chrome and Opera in scale of 48px (muliple sizes can be added)
<link rel="icon" sizes="192x192" href="icon.png">
IE Tiles :: 4 sizes. Listed are standard size. 1.8 times is better.
<meta name="application-name" content="CenturyCutCook" /> <meta name="msapplication-TitleColor" content="#009900" /> <meta name="msapplication-sqaure70x70logo" content="icon_smalltitle.png"> <meta name="msapplication-sqaure150x150logo" content="icon_mediumitle.png"> <meta name="msapplication-sqaure310x150logo" content="icon_widetitle.png"> <meta name="msapplication-sqaure310x310logo" content="icon_largetitle.png">
Apple Icon :: Default 60x60
<link rel="apple-touch-icon" href="touch-icon-iphone.png"> <link rel="apple-touch-icon" sizes="76x76" href="touch-icon-ipad.png"> <link rel="apple-touch-icon" sizes="120x120" href="touch-icon-iphone-retina.png"> <link rel="apple-touch-icon" sizes="152x152" href="touch-icon-ipad-retina.png">
Apple :: Startup Image. By default, Safari shows a blank screen during load time and after multiple loads a screenshot of the previous state of the app. Base Size :: 320x480. But different devices ask for different sizes. Follow this to implement
<link rel="apple-touch-startup-image" href="/startup.png"> <!-- iPad retina portrait startup image --> <link href="https://placehold.it/1536x2008" media="(device-width: 768px) and (device-height: 1024px) and (-webkit-device-pixel-ratio: 2) and (orientation: portrait)" rel="apple-touch-startup-image"> <!-- iPad retina landscape startup image --> <link href="https://placehold.it/1496x2048" media="(device-width: 768px) and (device-height: 1024px) and (-webkit-device-pixel-ratio: 2) and (orientation: landscape)" rel="apple-touch-startup-image"> <!-- iPad non-retina portrait startup image --> <link href="https://placehold.it/768x1004" media="(device-width: 768px) and (device-height: 1024px) and (-webkit-device-pixel-ratio: 1) and (orientation: portrait)" rel="apple-touch-startup-image"> <!-- iPad non-retina landscape startup image --> <link href="https://placehold.it/748x1024" media="(device-width: 768px) and (device-height: 1024px) and (-webkit-device-pixel-ratio: 1) and (orientation: landscape)" rel="apple-touch-startup-image"> <!-- iPhone 6 Plus portrait startup image --> <link href="https://placehold.it/1242x2148" media="(device-width: 414px) and (device-height: 736px) and (-webkit-device-pixel-ratio: 3) and (orientation: portrait)" rel="apple-touch-startup-image"> <!-- iPhone 6 Plus landscape startup image --> <link href="https://placehold.it/1182x2208" media="(device-width: 414px) and (device-height: 736px) and (-webkit-device-pixel-ratio: 3) and (orientation: landscape)" rel="apple-touch-startup-image"> <!-- iPhone 6 startup image --> <link href="https://placehold.it/750x1294" media="(device-width: 375px) and (device-height: 667px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image"> <!-- iPhone 5 startup image --> <link href="https://placehold.it/640x1096" media="(device-width: 320px) and (device-height: 568px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image"> <!-- iPhone < 5 retina startup image --> <link href="https://placehold.it/640x920" media="(device-width: 320px) and (device-height: 480px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image"> <!-- iPhone < 5 non-retina startup image --> <link href="https://placehold.it/320x460" media="(device-width: 320px) and (device-height: 480px) and (-webkit-device-pixel-ratio: 1)" rel="apple-touch-startup-image">
7.12.9. rel="noopener" html:link:rel:noopener
HTML link to another domain with
target="_blank", better add this so thatwindow.openerobject doesn't exist on the new page in new tab<a href="anotherdomain.gg" target="_blank" rel="noopener">...</a>
7.13. Autofill, Input Types
https://developers.google.com/web/updates/2015/06/checkout-faster-with-autofill
Autofill hint set :: the address or contact info type
- autocomplete="shipping locality"
- "shipping" or "billing" meaning the field is part of the shipping address or contact information.
These apply to telephone, emails and instant messaging
- home, work, mobile, fax, pager
section-* may be added with autofill hint set. The whole thing is called autofill scope.
Autofill fields :: have to match <input> type e.g. text, select, etc.
Wrap <input> inside <form> in order for autocomplete to work in Chrome.
[section-](optional) [shipping|billing](optional) [home|work|mobile|fax|pager](optional) [autofill field name]
<label for="frmNameCC">Name on card</label> <input name="ccname" id="frmNameCC" required placeholder="Full Name" autocomplete="cc-name"> <label for="frmCCNum">Card Number</label> <input name="cardnumber" id="frmCCNum" required autocomplete="cc-number"> <label for="frmCCCVC">CVC</label> <input name="cvc" id="frmCCCVC" required autocomplete="cc-csc"> <label for="frmCCExp">Expiry</label> <input name="cc-exp" id="frmCCExp" required placeholder="MM-YYYY" autocomplete="cc-exp"> <label for="frmCityS">City</label> <input name="ship-city" required id="frmCityS" placeholder="New York" autocomplete="shipping locality"> <fieldset> <legend>Ship the blue gift to...</legend> <p> <label> Address: <textarea name=ba autocomplete="section-blue shipping street-address"></textarea> </label> <p> <label> City: <input name=bc autocomplete="section-blue shipping address-level2"> </label> <p> <label> Postal Code: <input name=bp autocomplete="section-blue shipping postal-code"> </label> </fieldset> <fieldset> <legend>Ship the red gift to...</legend> <p> <label> Address: <textarea name=ra autocomplete="section-red shipping street-address"></textarea> </label> <p> <label> City: <input name=rc autocomplete="section-red shipping address-level2"> </label> <p> <label> Postal Code: <input name=rp autocomplete="section-red shipping postal-code"> </label> </fieldset>
7.13.1. Input Types
- url
- validate
http://,ftp://ormailto: - tel
- don't validate
- may validate? May use attribute
multiple - (no term)
- search
- number
- iOS requires pattern="\d*" to show the numeric keyboard
- (no term)
- range
- (no term)
- datetime-local
- (no term)
- date
- (no term)
- time
- (no term)
- month
- (no term)
- color
Provide suggestions for text field
<label for="frmFavChocolate">Favorite Type of Chocolate</label> <input type="text" name="fav-choc" id="frmFavChocolate" list="chocType"> <datalist id="chocType"> <option value="white"> <option value="milk"> <option value="dark"> </datalist>
Validate form :: Constraint Validation API https://developers.google.com/web/fundamentals/design-and-ux/input/forms/
7.14. Special Characters css:content
- In CSS content attribute, you need to use CSS value
- Convert tool
- http://www.evotech.net/articles/testjsentities.html
- Decimal value
▼- Hex value
▼- CSS Value (Hex)
\25BCli:before {content:'\25BC';}- JS Value (Hex)
\u25BCalert('\u25BC is HTML entity ▼')
- Numeric character reference
&#nnnn;Unicode code point decimal form
- Can be more than 4 digits
- ASCII
- https://theasciicode.com.ar/
- Originally only 0 to 127, 128 characters, 7 bits. Later gets extended to 0 to 255, 256 characters, 8 bits
&#xhhhh;- Unicode code point hexadecimal form
x- must be lowercase
h- mix case but uppercase is usual
- DTD Standards
Usually are predefined in markup languages using DTD
- HTML5 does not allow to extend or add more character entity references
- XML now is in XSD (XML Schema) but XSD also follows part of DTD. Can add more character entity references
- XML has 5 predefined entities, which usually need to be escaped inside an XML tag (as attribute or content)
- &
&- <
<- >
>- '
'- "
"
- e.g.
is a space. Case-sensitive - https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references
- Unicode
UTF-8 and UTF-16 are variable length encodings
- UTF-8, a character may occupy a minimum of 8 bits
The original 128 ASCII characters are the first 128 characters of UTF-8 (from 0 to 127)
- No logic on converting ASCII (128 to 255) to Unicode decimal form
- UTF-16, a character length starts with 16 bits
- UTF-32 is a fixed length encoding of 32 bits
On Windows
M-nnnntypes a Unicode character using the decimal form
nnnnhas to input using the number pad, leading zeroes can be skipped when typing00da M-x- In some software, it will convert to
ÚUnicode character - (no term)
- Works if the font supports the character
- https://en.wikipedia.org/wiki/List_of_Unicode_characters
- http://www.i18nguy.com/markup/ncrs.html
- https://www.utf8-chartable.de/unicode-utf8-table.pl
- http://unicode.org/charts/
✓- Checkmark heavy
✔- Checkmark light
🗸- Checkmark with square
☑
‑- Quotation mark
- Left Double Quotation Mark “
“- Right Double Quotation Mark ”
”- Left Single Quotation Mark ‘
‘- Right Single Quotation Mark (including English possessives and contractions) ’
’- Closing/Right angle quotation mark »
- css is
\00BB»
···CSS is\00B7- https://www.alt-codes.net/triangle-symbols
- ▼
▼\25BC\u25BC- ▾
▾\25BE\u25BE
7.15. Geolocation
Chrome v50+ only supports geolocation for https
7.15.1. .getCurrentPosition
<button onclick="getLocation()">Get Location</button>
<p id="demo"></p>
<script>
var x = document.getElementById("demo");
function getLocation() {
if (navigator.geolocation) {
navigator.geolocation.getCurrentPosition(showPosition, showError);
}
else {
x.innerHTML = "Not supported";
}
}
function showPosition(position) {
x.innerHTML = "Latitiude" + position.coords.latitude +
"Longitude: " + position.coords.longitude;
}
function showError(e) {
switch (e.code) {
case: e.PERMISSION_DENIED:
break;
case: e.POSITION_UNAVAILABLE;
case: e.TIMEOUT:
case: e.UNKNOWN_ERROR:
}
}
</script>
7.15.2. watchPosition
Same as getCurrentPosition but continuous track.
7.16. Storage
if (typeof(Storage) !== "undefined") {
// Use localStorage or sessionStorage
// Store. value is always string
localStorage.setItem("key","value");
console.log(localStorage.getItem("key"));
localStorage.clickcount = Number(localStorage.clickcount) + 1;
console.log(localStorage.key);
}
else {
// Not supported
}
7.17. Application Cache
<html manifest="demo.appcache"
.appcache file should be served in text/cache-manifest media type
CACHE MANIFEST # Above should be first line # Files that should be cached /theme.css NETWORK # Files will not be cached login.asp # An asterisk indicates all other files require an internet connection * FALLBACK /html/ /offline.html # offline.html should be served in place of all files in /html/ directory # if internet is not available # Change the manifest file to update cache
7.18. ARIA html:ARIA
7.18.1. role attribute
- separator
- BS4
.dropdown-divider - group
- BS4
.btn-group - button
<a>withouthref- search
<form>define purpose- presentation
- element does not need to be mapped to the accessibility API
7.18.2. aria-* attributes
aria-label- Label or name is used interchangably. e.g. Inside
<button>where label text can't be displayed aria-labelledby- Labelled by other element(s). Refer to other elements using their ids.
- Can be nested
aria-labelledby="parentID child1ID" - Say parentID element contains text Men's Shirts and childID element contains text Shop Now, then the resulted name of the element used
aria-labelledbyhas a name "Men's Shirts Shop Now" - it can even refer to itself
- Can be nested
- (no term)
aria-hidden="true"<h1 aria-label="Hello World"> <div class="grid" aria-hidden="true"> <!-- the following are not incluced in any accessibility device --> <span>H</span> <span>e</span> <span>l</span> <span>l</span> <span>o</span> <span>w</span> <span>o</span> <span>r</span> <span>l</span> <span>d</span> </div> </h1>
7.19. Display price
<span class="price-49-cents">4.9</span> <style> .price-49-cents { font-size:70px; } .price-49-cents:after { content; "\00A2 \A per kWh"; white-space: pre; /* wrap when line break is encountered */ font-size:16px; /* line-height:16px; probably don't need it */ vertical-align:top; display:inline-block; } </style>
7.20. Progressive Web App
7.21. AMP google:amp
- https://www.ampproject.org/learn/about-how/
- JavaScript and asset files are inside iframe. Only async js is allowed
- Predefine size in HTML for assets/resources, ads or iframes
- Custom script will eventually have a custom tag (iframe) to be loaded in
- 50kb max
- The above fonts start to download immediately
- Only certain animations are allowed (transform and opacity)
- Resources are loaded as late as possible (lazy-load, above the fold), but prefetched as early as possible
- Many AMP features require HTTPS
- inside
<!doctype html> - https://search.google.com/test/amp
- http://localhost:8000/released.amp.html#development=1
- https://playground.amp.dev/
- https://validator.ampproject.org/
- AMP Validator Chrome Extension
7.21.1. Search Engine Discovery
- https://www.ampproject.org/docs/fundamentals/spec
<link rel="amphtml" href="https://www.example.com/url/to/amp/document.html"><link rel="canonical" href="https://www.example.com/url/to/full/document.html">- If there's only one page and the page is AMP
<link rel="canonical" href="https://www.example.com/url/to/amp/document.html">
7.21.2. AMP JS Library
<script async src="https://cdn.ampproject.org/v0.js"></script>
7.21.3. Boilerplate
<style amp-boilerplate>body{-webkit-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-moz-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-ms-animation:-amp-start 8s steps(1,end) 0s 1 normal both;animation:-amp-start 8s steps(1,end) 0s 1 normal both}@-webkit-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-moz-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-ms-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-o-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}</style><noscript><style amp-boilerplate>body{-webkit-animation:none;-moz-animation:none;-ms-animation:none;animation:none}</style></noscript>
7.21.4. amp-custom amp:amp-custom
<style amp-custom> /* any custom style goes here */ body { background-color: white; } amp-img { background-color: gray; border: 1px solid black; } </style>
7.21.5. amp-image amp:amp-image
- Basics
- Built-in element, no need to import another AMP runtime
<amp-img src="/static/samples/img/amp.jpg" alt="AMP" width="475" height="268" layout="responsive"> <noscript> <img src="/static/samples/img/amp.jpg" width="475" height="268" alt="AMP"> </noscript> </amp-img> <div> <amp-img src="https://unsplash.it/300/200/?image=100" width="300" height="200" [src]="myImageUrl"> </amp-img> <button on="tap:AMP.setState({ myImageUrl: 'https://unsplash.it/300/200/?image=200' })">Change image</button> </div>
7.21.6. amp-bind amp-bind-macro amp:amp-bind
7.21.6.1. amp-bind
- Refer to amp:event
<script async custom-element="amp-bind" src="https://cdn.ampproject.org/v0/amp-bind-0.1.js"></script> <div> <div hidden [hidden]="hideGreeting">Hello World</div> <button on="tap:AMP.setState({ hideGreeting: false })">Show greeting</button> </div> <div> <div>Hello <span [text]="myText">World</span></div> <button on="tap:AMP.setState({ myText: 'AMP' })">Change text</button> </div> <div> <div class="background-red" [class]="myClass">Hello World</div> <button on="tap:AMP.setState({ myClass: 'background-green' })">Change class</button> </div> <div> <amp-img src="https://unsplash.it/400/200" width="200" [width]="myImageDimension.width" height="100" [height]="myImageDimension.height"> </amp-img> <button on="tap:AMP.setState({ myImageDimension: { width: 400, height: 200 } })"> Change size </button> </div> <div> <select on="change:AMP.setState({ option: event.value })"> <option value="0">No selection</option> <option value="1">Option 1</option> <option value="2">Option 2</option> </select> <div hidden [hidden]="option != 1"> Option 1 </div> <div hidden [hidden]="option != 2"> Option 2 </div> </div> <!-- initialize state --> <div> <amp-state id="myText"> <script type="application/json"> "World" </script> </amp-state> <div>1. Hello <span [text]="undefinedText">World</span></div> <div>2. Hello <span [text]="myText">World</span></div> <button on="tap:AMP.setState({ myText: 'AMP' })">Change state</button> </div> <!-- remote state --> <!-- empty AMP.setState() triggers all amp-bind's to evaluate --> <div> <amp-state id="myRemoteState" src="/static/samples/json/websites.json"></amp-state> <amp-list layout="fixed-height" height="0" [height]="18 * myRemoteState.items.length" [src]="myRemoteState.items" binding="no"> <template type="amp-mustache"> <div><a href="{{url}}">{{title}}</a></div> </template> </amp-list> <button on="tap:AMP.setState({})">Show websites</button> </div> <!-- refresh one state --> <div> <amp-state id="currentTime" src="/documentation/examples/api/time"></amp-state> <button on="tap:currentTime.refresh"> Refresh </button> <div [text]="currentTime.time"></div> </div> <!-- push state --> <!-- browsser's back button --> <div> <amp-state id="count"> <script type="application/json"> 1 </script> </amp-state> <div>Item <span [text]="count">1</span></div> <button on="tap:AMP.pushState({ count: count + 1 })">Increase count</button> </div> <!-- debounce input events --> <div> <amp-state id="name"> <script type="application/json"> "?" </script> </amp-state> <input id="name-input" placeholder="Enter a name" on="input-throttled:AMP.setState({ name: event.value })"> <div>Hello <span [text]="name">?</span></div> </div>
7.21.6.2. amp-bind-macro
<div> <amp-bind-macro id="circleArea" arguments="radius" expression="3.14 * radius * radius" /> <input type="number" min="0" max="100" value="0" on="input-throttled:AMP.setState({ radius: event.value })"> <div> The circle has an area of <span [text]="circleArea(radius)">0</span>. </div> </div>
7.21.7. amp-carousel amp:amp-carousel
Basics
<script async custom-element="amp-carousel" src="https://cdn.ampproject.org/v0/amp-carousel-0.1.js"></script> <amp-carousel height="300" layout="fixed-height" type="carousel"> <amp-img src="/static/samples/img/image1.jpg" width="400" height="300" alt="a sample image"></amp-img> <amp-img src="/static/samples/img/image2.jpg" width="400" height="300" alt="another sample image"></amp-img> <amp-img src="/static/samples/img/image3.jpg" width="400" height="300" alt="and another sample image"></amp-img> </amp-carousel>
- https://amp.dev/documentation/examples/multimedia-animations/image_galleries_with_amp-carousel/
- Attributes
- type
- carousel
- may use
autoplayat d:5000 milliseconds, usedelayto override
- Style buttons by targeting
.amp-carousel-button-prevand.amp-carousel-button-next
7.21.8. amp-video amp:amp-video
Basics
<script async custom-element="amp-video" src="https://cdn.ampproject.org/v0/amp-video-0.1.js"></script> <amp-video width="480" height="270" src="/static/samples/video/tokyo.mp4" poster="/static/samples/img/tokyo.jpg" layout="responsive" controls> <div fallback> <p>Your browser doesn't support HTML5 video.</p> </div> <source type="video/mp4" src="/static/samples/video/tokyo.mp4"> <source type="video/webm" src="/static/samples/video/tokyo.webm"> </amp-video>
7.21.9. amp-position-observer amp:amp-position-observer
- https://amp.dev/documentation/components/amp-position-observer/
- Use it with amp:amp-animation and several video players are the only components that allow low trust events to trigger their actions. Using it alone does not do anything
- It dispatches events
- enter
- can be tied to
startaction of amp:amp-animation - exit
- can be tied to
pauseaction of amp:amp-animation scroll:<Position in Viewport As a Percentage>- can be tied to
seekToaction of amp:amp-animation
- Attributes
- target
- ID of the element to observe. If not specified, the parent of this tag is used as the target
- intersection-ratios
- d:0 between 0 and 1. How much of the target should be visible in the viewport before
amp-position-observertriggers any events- 0
- trigger
enterwhen a single pixel of the target comes into vp and triggerexitwhen the very last pixel goes out of the vp - 0.5
- trigger
enteras soon as 50% of the target comes into vp and triggerexitwhen less than 50% is in the vp - 1
- trigger
enterwhen fully visible and trigger event when a single pixel goes out of vp - 0 1
- trigger
enterwhentop=0visible and trigger event whenbottom=1
- viewport-margins
- 100px
- shrink the vp by 100px from the top and 100px from the bottom
- 25vh
- shring the vp by 25% from the top and 25% from the bottom. Only consider the middle 50% of the viewport
- 100px 10vh
- shrink the vp by 100px from the top and 10% from the bottom
- once
- only trigger the
enterandexitonce
<script async custom-element="amp-position-observer" src="https://cdn.ampproject.org/v0/amp-position-observer-0.1.js"></script>
7.21.10. amp-animation amp:amp-animation
<script async custom-element="amp-animation" src="https://cdn.ampproject.org/v0/amp-animation-0.1.js"></script> <script async custom-element="amp-position-observer" src="https://cdn.ampproject.org/v0/amp-position-observer-0.1.js"></script> <div class="parallax-image-window"> <amp-position-observer on="scroll:parallaxTransition.seekTo(percent=event.percent)" intersection-ratios="0" viewport-margins="15vh" layout="nodisplay"> </amp-position-observer> <a href="%%CLICK_URL_UNESC%%%%DEST_URL%%" target="_blank"> <amp-img id="parallaxImage" width="300" height="600" layout="responsive" src="%%VIEW_URL_UNESC%%%%FILE:JPG1%%" alt="Picture of an elephant"></amp-img> </a> </div> <amp-animation id="parallaxTransition" layout="nodisplay"> <script type="application/json"> { "duration": "1", "fill": "both", "direction": "normal", "animations": [{ "selector": "#parallaxImage", "keyframes": [ {"transform": "translateY(-600px)"} ] }] } </script> </amp-animation>
7.21.11. Actions and Events amp:event
- Basics
https://amp.dev/documentation/guides-and-tutorials/learn/amp-actions-and-events/
<input id="name-input" placeholder="Enter a name" on="input-throttled:AMP.setState({ name: event.value })">
- Syntax
eventName:targetId[.methodName[(arg1=value, arg2=value)]]- eventName
- multiple events
on="submit-success:lightbox1;submit-error:lightbox2"- multiple events for one action
on="tap:target1.actionA,target2.actionB"
- refer to amp:event:target
- is also called action
- could be
- unquoted
simple-value- quoted
"string value"or'string value'- bool
truefalse- numbers
111.1- event data
event.someDataVariableName
- eventName
- (no term)
- all elements events
- tap
- hide
- show
- toggleVisibility
- toggleClass(class=STRING, force=BOOLEAN)
- scrollTo(duration=INTEGER, position=STRING)
- focus
- (no term)
- input events
- input-throttled
- same as
changebut at most once every 100ms
7.21.11.1. Special targets amp:event:target
- Can be DOM id of an element
- https://amp.dev/documentation/guides-and-tutorials/learn/amp-actions-and-events/#special-targets
7.21.12. Layout & media queries amp:layout
- attribute
layoutfor all elements - https://amp.dev/documentation/guides-and-tutorials/develop/style_and_layout/control_layout/
- nodisplay
7.21.13. amp-analytics amp:amp-analytics
https://amp.dev/documentation/components/amp-analytics/
<script async custom-element="amp-analytics" src="https://cdn.ampproject.org/v0/amp-analytics-0.1.js"></script>- Attributes
type-
googleanalytics- refer to ga:amp
<amp-analytics type="googleanalytics" id="analytics1"> <script type="application/json"> { "vars": { "account": "UA-XXXXX-Y" }, "triggers": { "trackPageview": { "on": "visible", "request": "pageview" } } } </script> </amp-analytics>
- attribute config (inline config)
- remote config > inline config > vendor config based on attribute
typeconfig
7.21.14. AMPHTML ads
- https://amp.dev/documentation/guides-and-tutorials/start/create_amphtml_ad/create_shell/?format=ads
- https://amp.dev/documentation/examples/?format=ads
- Can be served on both AMP pages (via Ad Manager AMPHTML ad tag) and non-AMP pages (via Google Publisher Tag)
- Allowed AMP extensions and builtins
- amp:amp-accordion
- amp:amp-ad-exit
- amp:amp-analytics
- amp:amp-anim
- amp:amp-animation
- amp:amp-audio
- amp:amp-bind
- amp:amp-carousel
- amp:amp-fit-text
- amp:amp-font
- amp:amp-form
- amp:amp-img
- amp:amp-layout
- amp:amp-lightbox
- amp:amp-mraid
- amp:amp-mustache
- amp:amp-pixel
- amp:amp-position-observer
- amp:amp-social-share
- amp:amp-video
<!doctype html> <html ⚡4ads> <head> <meta charset="utf-8"> <title>My amphtml ad</title> <meta name="viewport" content="width=device-width,minimum-scale=1,initial-scale=1"> <!-- AMPHTML ads do not require a <link rel="canonical"> tag. --> <!-- runtime --> <script async src="https://cdn.ampproject.org/amp4ads-v0.js"></script> <!-- boilerplate --> <!-- rationale: hide body until AMP runtime is ready and then unhides --> <style amp4ads-boilerplate>body{visibility:hidden}</style> <!-- creative CSS is limited to 20KB --> <style amp-custom></style> <a target="_blank" href="https://amp.dev/documentation/examples/style-layout/banner_ad/index.html"> <amp-img src="https://amp.dev/static/samples/img/amp-300x250.jpg" width="300" height="250" layout="responsive" alt="a4a image"></amp-img> </a> </head> <body> </body> </html>
8. JavaScript
8.1. Loading Sequence js:defer js:async
- Refer to html:event:DOMContentLoaded
<script>tags without any attributes are downloaded together and executed immediately in order- defer
- Works when src is defined. Download together, execute in order and defer execution until before
DOMContentLoaded. The download and execute process do not block HTML parsing - async
- Works when src is defined. Download together, execute in any order as soon as they're downloaded and
DOMContentLoadeddoes not have to wait for the execution. The download process does not stop HTML passing, but the execution process does. If async scripts are downloaded and executed before DCL, it will still block HTML parsing async="false"- Works when src is defined. Download together, execute in order as soon as all are downloaded
- (no term)
- async is useful when the script files are not dependent on other files and do not have any dependencies themselves. Use it when you don't care exactly at which point the file is executed
- (no term)
- By default, all dynamically added scripts are async
- (no term)
For example, if you want to load 2 javascript files and execute them in order but after they are all downloaded, then
async="false"is needed[ '1.js', '2.js' ].forEach(function(src) { var script = document.createElement('script'); script.src = src; script.async = false; document.head.appendChild(script); });
8.2. External Script, document.write js:external script
Loading external script use this
[ 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/jquery-ui.min.js','/sites/all/themes/abc/js/interstitial.js' ].forEach(function(src) { var script = document.createElement('script'); script.src = src; script.async = false; document.head.appendChild(script); // another way var s = document.scripts[0]; s.parentNode.insertBefore(script, s); });
- refer to css:js insert
document.writeinside external script (asynchronously loaded) cannot modify DOM. Use these to modify the DOM. Refer to html:ElementObject.appendChild().insertBefore()var t = document.getElementById("childElement"); t.parentNode.insertBefore(newNode, t);
equivalent to jQuery
.html('html')- Get or set the HTML or XML markup contained within the element
- Set
remove all of the element's descendants and replace them with nodes constructed by parsing the HTML given in the string
<script>elements inserted using innerHTML do not execute when they are insertedonclickattributes are valid
- Refer to php:json_encode
- May use with array
var html = []; // var html = <?php echo $html; ?>; html.push(<?php echo $html; ?>); html.push(<?php echo $html2; ?>); // t.innerHTML = html; // html is converted to string with commas t.innerHTML = html.join('');
get and set text inside the element
- Get
- result is a plain text with all HTML tags stripped
- trigger reflow
- Set
- set text only. No HTML tag parsing
- https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent
- concatenation of the textContent of every child node, excluding comments and processing instructions
- Set textContent on a node removes all of the node's children and replaces them with a single text node with the given string value
- Refer to tool:html:document.write
https://support.jwplayer.com/customer/portal/articles/1406723#fndtn-method-1
- Refer to video:jw
<script src="//content.jwplatform.com/players/MEDIAID-PLAYERID.js"> </script>- If this is async loaded, the player will not show because it has
document.write <script src="//content.jwplatform.com/libraries/PLAYERID.js"> </script>
- And a container
<div id="myElement"></div>- Then dynamically load this
TEMPLATEIDis the Source file name for a video content in JW UI (under Sources).<script type="text/javaScript"> var playerInstance = jwplayer("myElement"); playerInstance.setup({ file: "//content.jwplatform.com/videos/MEDIAID-TEMPLATEID.mp4" mediaid: "xxxxYYYY" }); </script>
8.3. Initialization >
Always initialize variables at the top including vars in loops
var myObj = {}, myString = "", myArray = [], myBoolean = false, myRegex = /()/, myNumber = 0;
These values are considered as false
8.4. Variable Scope
8.4.1. Hoisting
- See 8.6.10
varvariables are hoisted to the top of their scope and initialized with a value ofundefined- Only function can create local scope
- This is called function-level scoping while PHP has block-level scoping, where new scope can be defined in block
- Without using
varorletto declare variables in any function (it's called implicitly declared) and that function runs, those variables are global
myFunction(); function myFunction() { carName = "Volvo"; // carName is actually global var carName2 = "Freightliner"; // carName2 is not global // Local var takes precedence when it's used inside a function // the inner var shadows the outer. } // Alternative way to get global var console.log(window.carName); // function and variable declaration are moved to the top of their // scope (global or local). It's called hoisting // which allows you to use them before they are declared // Functions are hoisted first and then variables // Function declarations have priority over variable declarations, but not variable assignments showState("one", {}, 2, 0.98); // output: Ready // function declaration // Pros :: Can be used before function is declared // Cons :: Can't conditionally declare a function function showState() { console.log("Ready"); // arguments[0] is "one" } // function expression/definition/assignment // Pros :: Can conditionally define a function // Cons :: Can't be used before it is defined var showState = function() { console.log("Idle"); }; showState(); // output: Idle // The function that is assigned to a variable doesn't have to be anonymous really.long.external.scoped.name = function shortcut(n) { // regression shortcut(n-1); // Pass it self as a callback someFunction(shortcut); }
8.4.2. let, const, var
var- A variable is defined by
varthen this variable is available to the closest function scope- When a
varis declared outside a function - global
- When a
varis declared inside a function block - function scoped
- When a
- Always js:initialize for variables declared by
var - See 8.4.1
- A variable is defined by
letandconstare block-scopedletdeclares a variable which is available to the block it is enclosed in- A block is anything inside
{}e.g.switch,for,if - Can only
letdeclare the same variable once in the same block- The same variable can be
letdeclaired inside a subblock and it will have new scope but not overwrite the upper scope variable
- The same variable can be
letdeclared variable must be declared before it is used or referenced- No 8.4.1
- Limitation about
var- Use
letin a loop, the variable can be re-binded for each iteration while
varin a loop, only the final assigned value is passed unless a closure is usedfor (var i = 1; i <= 5; i++) { setTimeout( function () { console.log(i); }, 1000 * i); // 5 5 5 5 5 setTimeout( function (x) { return function () { console.log(x); }; }(i) , 1000 * i); // 1 2 3 4 5 }
- Use
constis the same asletbut it's read only. Uppercase naming- The same
constvariable name can be redeclared to different value in sub block - If
constis an array or object, its child element or key/value is not protected by default- Which means you can push to
constarray or modify key/value
- Which means you can push to
constin the same block cannot be changed or redeclared or redeclared byletorvarconstcan be used in js:for…of and js:for…in loops but not the imperativeforlooplet scores = [75, 80, 95]; for (const score of scores) { console.log(score); } // throws error for (const i = 0; i < scores.length; i++) { // TypeError console.log(scores[i]); }
- The same
8.5. Closure, Javascript Singleton
8.5.1. Compare to PHP Closure php:closure
8.5.2. IIFE (Immediately Invoked Function Expression)
// function declaration function() { console.log('test') } // returns function(){ console.log('test') } (function() { console.log('test'); }) function myFunc() { // true except when var anInstance = new myFunc(); console.log(this === window); return "global function myFunc"; } // Immediately Invoked Function Expression (IIFE) Variation 1 var abc = (function() { console.log(this === window); return 'test'; })(); console.log(abc === "test"); // Or var abc = (function myFunc() { console.log(this === window); return 'test'; })(); console.log(abc === "test"); console.log(myFunc() === "global function myFunc"); // IIFE Variation 2 var abc = (function() { console.log(this === window); return "test"; }()); // Or var abc = (function myFunc() { return "test"; }()); console.log(abc === "test"); console.log(myFunc() === "global function myFunc"); // IIFE Variation 3 // ! eq. to +, -, or even ~. Any unary operator // returns true. because the function run returns nothing (undefined) !undefined === true // Don't use it for assigning value to a variable !function() { console.log(this === window); }(); // return false because IIFE returns 'test' and !'test' === false !function() { return 'test'; }(); // Don't do this. abc will be false var abc= !function() { return 'IIFE'; }(); // IIFE Variation 4 (only for assigning return value to a variable) var abc = function() { return 'test'; }(); // Or var abc = function myFunc() { return 'test'; }(); // IIFE Variation 5. Use parent's `this` inside the IIFE // See js:function:call function Person() { this.instanceProperty = 10; var varVar = "a"; let letVar = "b"; setTimeout(function() { console.log('Person setTimeout callback is not an arrow function'); console.log(this === window); console.log(varVar === "a"); console.log(letVar === "b"); }, 0); setTimeout(() => { console.log('Person setTimeout callback is an arrow function executed under the instance scope'); console.log(this !== window); console.log(this.instanceProperty === 10); console.log(varVar === "a"); console.log(letVar === "b"); ;(function() { console.log("IIFE with 'this' overriden"); console.log(this !== window); console.log(this.instanceProperty === 10); console.log(varVar === "a"); console.log(letVar === "b"); }.call(this)); !function() { console.log("IIFE without overriding 'this'"); console.log(this === window); console.log(varVar === "a"); console.log(letVar === "b"); }(); }, 0); }
8.5.3. Revealing Module Pattern
- It means to use function scope, IIFE and JavaScript Object Prototype to create public and private methods
- More on 8.5.7
- Check another pattern 8.17
var Singleton = function () { var privateCounter = 0; function changeBy(val) { privateCounter += val; } // private methods function sayHello() { console.log('Hello'); } // public methods return { sayHello: sayHello, increment: function () { changeBy(1); }, callBothPrivatePublicMethods: function () { changeBy(1); this.increment(); }, getValue: function () { return privateCounter; }, callbackExample: function () { [].forEach.call( document.getElementsByClassName('my-class'), function (ele) { changeBy(1); } ) }, }; }(); // Hello Singleton.sayHello(); Singleton.increment(); // 1 console.log(Singleton.getValue()); Singleton.callbackExample(); // 4 console.log(Singleton.getValue()); Singleton.callBothPrivatePublicMethods(); // 6 console.log(Singleton.getValue()); // Non-singleton // Expose module as global variable var Module = function () { // Inner logic function sayHello() { console.log('Hello'); } // Expose API return { sayHello: sayHello } }; // instantiate var module1 = Module(); // Or // var module1 = new Module(); module.sayHello();
8.5.4. Semi-colon
// Use this IIFE format if semi colon may or may not be enforced lilitest = function() { console.log('lilitest'); } // no ; !function() { console.log('IIFE'); }() // no ;, returns true. lilitest is not run. Perfect! lilitest = function() { console.log('lilitest'); } // no ; !function() {}(); // with ;, returns true. lilitest is not run. Perfect! lilitest = function() { console.log('lilitest'); }; // with ; !function() {}(); // with ;, returns true. lilitest is not run. Perfect! lilitest = function() { console.log('lilitest'); }; // with ; !function() {}() // no ;, returns true. lilitest is not run. Perfect! // For this IIFE format, always add ; to each line otherwise expect unexpected results lilitest = function() { console.log('lilitest'); } // no ; (function() {console.log('IIFE');})() // no ;, lilitest is run! Error. NOT GOOD lilitest = function() { console.log('lilitest'); } // no ; (function() {})(); // with ;, lilitest is run! NOT GOOD lilitest = function() { console.log('lilitest'); }; // with ; (function() {})() // no ;, lilitest is not run! Perfect! lilitest = function() { console.log('lilitest'); }; // with ; (function() {})(); // with ;, lilitest is not run! Perfect! lilitest = function() { console.log('lilitest'); }; // with ; (function() {})(); // with ;, lilitest is not run. Perfect! // For this IIFE format, always add ; to each line otherwise expect unexpected results lilitest = function() { console.log('lilitest'); } // no ; (function() {console.log('IIFE');}()) // no ;, lilitest is run! NOT GOOD lilitest = function() { console.log('lilitest'); } // no ; (function() {console.log('IIFE');}()); // with ;, lilitest is run! NOT GOOD lilitest = function() { console.log('lilitest'); }; // with ; (function() {console.log('IIFE');}()) // no ;, lilitest is not run! Perfect! lilitest = function() { console.log('lilitest'); }; // with ; (function() {console.log('IIFE');}()); // with ;, lilitest is not run! Perfect! // Variable assigned by the return of IIFE is fine with inconsistent semi-colons lilitest = function() { console.log('lilitest'); } // no ; lilitest2 = (function() {})(); // with ;, lilitest is not run. Perfect! lilitest = function() { console.log('lilitest'); } // no ; lilitest2 = (function() {})() // no ;, lilitest is not run. Perfect! lilitest = function() { console.log('lilitest'); } // no ; lilitest2 = function() {}(); // with ;, lilitest is not run. Perfect! lilitest = function() { console.log('lilitest'); } // no ; lilitest2 = function() {}() // no ;, lilitest is not run. Perfect!
8.5.5. Closure does not have to be defined as a result of immediately run
function outer() { var b = 10; function inner() { var a = 20; console.log(a+b); } return inner; } var X = outer();
8.5.6. Syntax, When to use closure?
- A closure gives access to an outer function's scope from an inner function
- Closures are created every time a function is created, at function creation time
- Closure is a function bundled together (enclosed) with references to its surrounding state (lexical environment)
- When you don't want to change the variables in outer scope where the closure is in
- The outer scope can be global or any place where the closure is in
- When you want to run the function later
- When you want to run the function later by providing just reference to other function
- When you want to run the function later by providing initial state
- When you want to do regression, don't want to affect outer scope and need more debugging
(function() { // Codes here runs immediately })();
(function($) { // Use jQuery with the shortcut console.log($.browser); $(document).ready(function($) { // document ready } ); }(jQuery)); // Another form jQuery(document).ready(function($) { // Use $ now });
8.5.7. Singleton
- It's also called Revealing Module Pattern
- See js:oop for instantiation
var Managers = {}; //namespace // Singleton object CssManager // `Managers` and singleton object `CssManager` has to be defined before they are used Managers.CssManager = (function () { //region Private Space // All variables and methods are protected from the global scope // To initialize, assign aliases of global properties/methods/functions/objects for this Singleton var doc = document; // Private members can have the same name as in the object and they are referred diffrently inside the object // - privateAndMember(); // - this.publicMember(); var setAttributes = function (element, attributes) { for (attribute in attributes) { element[attribute] = attributes[attribute]; } } //endregion return { addStyleSheet: function (id, url) { // global variable `document` can be used, or use alias `doc` as defined above in Private Space var newStyleSheet = document.createElement("link"); // call private member setAttributes(newStyleSheet, { rel: "stylesheet", type: "text/css", id: id, href: url }); document.getElementByTagName("head")[0].appendChild(newStyleSheet); }, removeStyleSheet: function (id) { var currentStyleSheet = document.getElementById(id); if (currentStyleSheet) { currentStyleSheet.parentNode.removeChild(currentStyleSheet); } }, swapStyleSheet: function (id, url) { // call other public member this.removeStyleSheet(id); this.addStyleSheet(id, url); } } })();
8.5.8. Regression
This provides you more debugging info
var charsInBody = (function counter(elm) { // Notice :: this is not an anonymous function as we see before in a closure! if (elm.nodeType == 3) { return elm.nodeValue.length; } var count = 0; for (var i = 0, child; child = elm.childNodes[i]; i++) { count += counter(child); } return count; })(document.body);
8.6. Types, typeof, constructor
8.6.1. Number
- Used in
- js:Math
IsNaN- true if it is not a number
var foo = "55"; var myNumber = Number(foo); Number(' 55 ') === Number('55.0') === 55 Number.isInteger(Number(' 55.00 ')) === true if (!isNaN(myNumber)) { console.log("It is a number"); // foo is a number } console.log( Number.parseFloat(x).toFixed(2) ); // round up sometimes not working using toFixed.. let decimals = 2; console.log( Number(Math.round(value +'e'+ decimals) +'e-'+ decimals).toFixed(decimals) ); // use this to round function isInt(value) { return !isNaN(value) && (function(x) { return (x | 0) === x; })(parseFloat(value)) } isInt(42) // true isInt("42") // true isInt(4e2) // true isInt("4e2") // true isInt(" 1 ") // true isInt("") // false isInt(" ") // false isInt(42.1) // false isInt("1a") // false isInt("4e2a") // false isInt(null) // false isInt(undefined) // false isInt(NaN) // false
8.6.2. Boolean
trueorfalse. NOTTRUEnotFALSE""0falsenullundefined
8.6.3. undefined undefined or has not been assigned a value
// Variable let name; if (name === undefined) { console.log("This is undefined"); } // Object let user = { name: "John Doe", age: 14 }; if (user.hobby === undefined) { console.log("This is undefined"); } // Array let scores = [12, 34, 66, 78]; if (scores[10] === undefined) { console.log("This is undefined"); } if(typeof user.hobby === "undefined"){} if(typeof scores[10] === "undefined"){} if(typeof name === "undefined"){}
8.6.4. null - a special type of object
8.6.5. String
- See 8.6.1 to convert a string to number
s.length; s.charAt(s.length-1); s.charCodeAt(s.length-1); // UTF-16 code unit value var myString = String.fromCharCode(65,66,67); // 'ABC' s.indexOf("Paul"); // Case sensitive. Starts from the beginning. -1 not found s.lastIndexOf("Paul"); // Case sensitive. Starts from the end s.substr(0,4); // From start to 4th index s.substring(0,4); // From start and take 4 characters. If stop is omitted, take the rest s.slice(0,4); // Same as substring but start and stop can be both or either negative // If start is negative, start from the end of string // If stop is negative, stop to (s.length - 1) - Math.abs(s) s.toLowerCase(); s.toUpperCase(); // Case insenstive comparison // Don't use string1 == string2 // Use below for comparing UTF8 if ( string1.toUpperCase() === string2.toUpperCase() ) {} // null converted to string is 'null' var abc = null; var abcDisplay = 'abc: ' + abc; console.log(abcDisplay); // "abc: null"
8.6.5.1. Padding 0's to left
function padLeft(nr, n, str){ return Array(n-String(nr).length+1).join(str||'0')+nr; } var mystring = 3; console.log(padLeft(mystring,3)); // 003 console.log(padLeft(mystring,3,'z')); // zz3
8.6.5.2. Template Literal
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals
Basics
// Template literal. Since js:es6 console.log(`This is ${myVar} times easier!`); let a = 5; let b = 10; console.log(`Fifteen is ${a + b} and not ${2 * a + b}.`); // need to escape backtick console.log(`Escape \``); // multi lines console.log(`string text line 1 string text line 2`); // "string text line 1 // string text line 2"
Inside an HTML's
<script>tag, and there's</script>inside it, no matter if it's a string in quotes, apostrophes, or even a template literal, it will always close the script tag. Escape it:<script> var str = ` <div> </div> <script> <\/script> ` var pre = document.createElement('pre') pre.textContent = str document.body.appendChild(pre) </script>
8.6.6. Array
8.6.6.1. Array.xxx() vs Array.prototype.xxx()
Array.xxx() is static method while Array.prototype.xxx() is instance method
8.6.6.2. DOM elements HTMLCollection
HTMLCollectionis not an array. Need to useArray.prototype.reduce.call(eles, function(a, b) {}, initialValue)[].reduce.call(eles, function (a,b) {}, initialValue)
8.6.6.3. Numeric array key exists
var a = [1,2]; if (a[8] !== undefined) { console.log(a[8]); }
8.6.6.4. Variable is a non-empty array
var a = { xyz: "abc" }; var myVar = a.abc; console.log((typeof notDeclared === "undefined") === true); console.log((Array.isArray(myVar) && myVar.length) === false); // Use this shortcut if type does not matter console.log(myVar?.length === undefined); console.log(a.abc?.length === undefined); console.log(a.xyz?.length === 3);
8.6.6.5. array.indexOf(), array.prototype.includes()
array.indexOf()
arr.indexOf(searchElement[, fromIndex]) var a = [1,2,3]; a.indexOf(4); // -1 a.indexOf(1); // 0 (['joe', 'jane', 'mary'].indexOf('jane') >= 0; // true
- All browsers support
===
-
arr.includes(valueToFind[, fromIndex]) ['joe', 'jane', 'mary'].includes('jane'); // true
- IE not supported
8.6.6.6. Array.prototype.pop(), Array.prototype.shift(), array.push(), array.unshift()
a.pop(); // remove last element a.shift(); // remove fist element, mutable a.push("Z"); // add to the end of array. return new length a[a.length]="Z"; // also append to last a.unshift("Z"); // add to the beginning of array. return new length
8.6.6.7. array.concat() merge without altering nor removing dups, return a new array
var a = new Array("P", "C", "S"); var b = new Array(31,29,34); a = a.concat(b); // Join arrays P, C, S, 31, 29, 34 // remove dup function removeDuplicate(arr){ var exists ={}, outArr = [], elm; for(var i =0; i<arr.length; i++){ elm = arr[i]; if(!exists[elm]){ outArr.push(elm); exists[elm] = true; } } return outArr; } // e.g. removeDuplicate([1,3,3,3,1,5,6,7,8,1]); // = [1, 3, 5, 6, 7, 8] // if you want mutation instead of creating a new array var a = ['a','b']; var b = ['a', 'x', 'y']; a.push.apply(a,b); // cross-browser //Array.prototype.push.apply(a,b); // cross-browser // a.push(a, ...b); // NodeJS where ES6/ES2015 is supported console.log(a); // ["a", "b", "a", "x", "y"]
8.6.6.8. array.slice() vs array.splice()
- Use js:array:filter to remove values from array instead if mutation is not desired
Array.prototype.slice([start[, end]]) // not mutated // start :: from 0 // end :: from 0. Up to this index but not including var a = [1,2,3]; var slice = a.slice(1,2); // [2] let arrDeletedItems = array.splice(start[, deleteCount[, item1[, item2[, ...]]]]) // mutated! // start :: from 0 // deleteCount :: An integer indicating the number of elements in the array to remove from start. If deleteCount is omitted, or if its value is equal to or larger than array.length - start (that is, if it is equal to or greater than the number of elements left in the array, starting at start), then all the elements from start to the end of the array will be deleted. // item1, item2, ... Optional :: The elements to add to the array, beginning from start. If you do not specify any elements, splice() will only remove elements from the array. var deletedItems = a.splice(1,2); // deletedItems = [2,3] and a becomes [1] // remove values in array var arr = [1, 2, 3, 4, 5, 5, 6, 7, 8, 5, 9, 0]; for( var i = 0; i < arr.length; i++){ if ( arr[i] === 5) { arr.splice(i, 1); i--; } } //=> [1, 2, 3, 4, 6, 7, 8, 9, 0]
8.6.6.9. array.join(), array.toString(), array.valueOf()
var join = a.join(","); // "P,C,S" a.toString() // also "P,C,S" a.valueOf(); // also "P,C,S". Convert to primitive
8.6.6.10. array.sort(), array.reverse()
var sort = a.sort(); // sort as strings. case sensitive: C, P, S. Mutable function numericSort(anArray) { anArray.sort(function(a,b) { return a - b; // ascending order // return a.year - b.year; if it's an obj }); return anArray; } function randomSort(anArray) { anArray.sort(function(a, b) {return 0.5 - Math.random();}); return anArray; } var reverse = a.reverse(); // S, C, P
8.6.6.11. array loop: for, array.forEach()
for (var i = 0; i < fruits.length; i++) { fruits[i]; }
fruits.forEach(function(i) { console.log(i); });
8.6.6.12. array.map(), array.filter()
- Do something while each item is iterated, join returned values into an array
- Use js:array:splice to remove values from array if mutation is desired
var result = a.map(function(i) { return i.toUpperCase(); }); // result is a new array console.log(a); // a is not affected! // array.filter // not mutated a = a.filter(function(i) { if ( true then remove condition ) { return false; } else { return true; // keep } }); var array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0]; var filtered = array.filter(function(value, index, arr){ return value > 5; }); //filtered => [6, 7, 8, 9] //array => [1, 2, 3, 4, 5, 6, 7, 8, 9, 0]
8.6.6.13. array.every(), array.some()
- See 2
// arr.every(callback(element[, index[, array]])[, thisArg]) a.every(function(i) { return i[0] === "a"; }); // return true only if all returns are true // If array is empty, it returns true. Does not matter what condition a.some(function(i) { return i[0] === "a"; }); // return true if at least one return is true
8.6.6.14. Array.prototype.reduce()
- The method executes a provided reducer function on each element of the array, resulting in a single output value
- See php:array_reduce
arr.reduce(callback( accumulator, currentValue[, index[, array]] )[, initialValue])- See https://github.com/levonlee/jsSandbox/blob/master/array.html#L33
var sum = [0, 1, 2, 3].reduce( function(a,b) { return a + b; }, 0 ); var sum2 = [...Array(4).keys()].reduce( // eq. for (let i = 0; i <= 3; i++) // eq. [0, 1, 2, 3] function(a, b) { return a + b; }, 0 ); // sum === sum2 === integer 6 var flattened = [[0, 1], [2, 3], [4, 5]].reduce( function(a,b) { return a.concat(b); }, [] ); // [0, 1, 2, 3, 4, 5]
array.reduce(callback[, initialValue])
- If initialValue is provided, it will start at index 0
- If not provided, it will start at index 1 and previousValue is the value at index 0 in the array
- callback has 4 arguments
- previousValue
- currentValue
- currentIndex
- the input array was called upon
8.6.7. Date
var date1 = new Date(); var date2 = new Date(77978); // milliseconds since 1970 01 01 GMT var unixTimestampFromPHP = parseInt( (date1.getTime()/1000).toFixed(0) ); date2 = new Date( unixTimestampFromPHP * 1000 ); var date3 = new Date(2017, 0, 31, 15, 35, 20, 20); // 2017 Jan 1 at 15:35:20 at 20 ms // ISO-8601 format works across browser var date5 = new Date('2011-04-11T10:20:30Z'); // Don't use any format other than the above. This does not guarantee to work var date4 = new Date("31 January 2016"); // Get and Set var dayofmonth = date4.getDate(); // setDate var dayofweek = date4.getDay(); // int, Sunday as 0. You can't set day var month = date4.getMonth(); // int, January as 0. setMonth var fullyear = date4.getFullYear(); // 4 digits. setFullYear var datestring = date4.toDateString(); // Current time zone. "Wed 31 Dec 2003" var currentDay = date4.getDate(); date4.setDate(currentDay + 28 ); // Plus 28 days // getHours(), getMinutes(), getSeconds(), getMilliseconds(), toTimeString() // setHours(), setMinutes(), setSeconds(), setMilliseconds() // Date difference var date1 = new Date(); var date2 = new Date('01-31-2016'); var timeDiff = date2.getTime() - date1.getTime(); // milliseconds in UTC var diff Days = timeDiff / (1000*3600*24); date1.getUTCFullYear() + '-' + String(date1.getUTCMonth() + 1).padStart(2, '0') + '-' + String(date1.getUTCDate()).padStart(2, '0') + '-' + String(date1.getUTCHours()).padStart(2, '0') + '-' + String(date1.getUTCMinutes()).padStart(2, '0') + '-' + String(date1.getUTCSeconds()).padStart(2, '0'); // YYYY-MM-DD-HH-mm-ss in UTC
8.6.7.1. Date to string
date4.toISOString(); // e.g. 2011-10-05T14:48:00.000Z // <<js:time:iso>> // timezone is always zero UTC offset. 24 or 27 (I haven't seen one..) characters. // Good for php:f:strtotime date4.toUTCString(); // UTC timezone. Sun, 23 Aug 2020 03:03:10 GMT date4.toGMTString(); // same as .toUTCString() date4.toString(); // browser's timezone. Sat Aug 22 2020 23:03:10 GMT-0400 (Eastern Daylight Time) date4.toLocaleString(); // use browser's default locale date4.toLocaleDateString(); // use browser's default locale // My browser's default locale is en-US // Use a specific locale date4.toLocaleString( 'en-US' ); // 8/22/2020, 10:01:19 PM date4.toLocaleDateString( 'en-US' ); // 8/22/2020 // Use options var options = { timezone: 'America/Toronto' }; date4.toLocalDateString( undefined, options ); // Use default locale, and use custom options options = { timezone: 'America/Toronto', month: 'short', day: 'numeric', year: 'numeric'}; date4.toLocaleDateString( 'en-US', options ); // Aug 22, 2020
- options
- all options
- Use options from Intl.DateTimeFormat
- default option
- is undefined for all options. If so,
year,monthanddayare assumed to be"numeric" - timezone
- IANA time zone names
8.6.7.2. Compare current date with a future date
function checkFuture(futureDateString) { var futureMS = new Date(futureDateString).getTime(); var currentMS = new Date().getTime(); if (currentMS >= futureMS) { return true; // current date is beyond future date } return false; } checkFuture("01/03/2018"); // mm/dd/yyyy
8.6.8. Math js:Math
Math.PI Math.abs(-101); Math.ceil(101.01); // 102 Math.floor(101.01); // 101 Math.round(101.01); Math.pow(10, 2); // 100 parseInt("10"); // 10 var itemCost = 9.99 * 1.075; // 10.73925 itemCost.toFixed(2); // 10.74
8.6.9. RegExp
8.6.9.1. search(), replace(), match()
var s = "Visit W3Schools!"; var n = s.search("W3Schools"); // 6. -1 if not found var all = s.match("i"); // return all matches var res = s.replace(/w3schools/i, "Microsoft"); var res = s.replace("W3Schools", "Microsoft");
8.6.9.2. RegExp
- Ways to define regex
var n = s.search(/w3schools/i); // Use forward slash. Return the first matched index var pattern = /w3schools/i; // Define a pattern in a variable // Need to escape \ with \\ var patternobj = new RegExp('w3schools','i'); // Define a pattern as a RegExp object var patternobj = new RegExp(/w3schools/,'i'); // Test a pattern pattern.test("Perform a test on this string"); // true or false // Return the first found text pattern.exec("Perform an exec on this string"); // null if not found. // Index every time pattern.exec() or pattern.test() is run, the pattern.lastIndex is advanced. Until .lastIndex gets to 0 // Return all matches s.match(pattern);
- Pattern and modifiers
- Syntax
/pattern/modifiers- (no term)
- Modifiers
- PHP PCRE Pattern Modifiers
- case-insensitive
- find all match
- multiline
- ungreedy
- the dot metacharacter matches all characters including newlines
(x|y)- one of x or y
- (no term)
Quantifier
n+ at least one of n n* zero or more n? zero or one n{X} \d{4} a 4-digit string n{X,Y} \d{3,4} a 3-to-4-digit string n{X,} \d{3,} a at-least-3-digit string n$ at the end ^n at the beginning (?=n) non greedy match
- Character Class or Character Set
- Brackets
- everything inside a pair of brackets are character class
- https://www.regular-expressions.info/refcharclass.html
- Refer to linux:regex:character class for special character classes that can only be used inside
[]- Some languages don't allow js:regex:metacharacter inside
[], so special character classes are the only choice
- Some languages don't allow js:regex:metacharacter inside
- Examples
[abc]- a character that is a, b or c
[^abc]- a character that is not a, b nor c
[0-9]- a digit
[^0-9]- a non-digit character
- Alphanumeric
[0-9a-zA-Z]
[base-[subtract]][a-z-[aeiuo]]- matches a single letter that is not a vowel
[base&&[intersect]][a-z&&[^aeiuo]]- matches a single letter that is not a vowel
- use backslash
\to escape- non-literal characters
^- as it's used in range and subtraction
]/
- Characters in special occasions
\- escape backslash itself
*- no need to escape inside
[] .- no need to escape inside
[]. Except in Nginx
- non-literal characters
- Metacharacter
- Some language can use metacharacters inside square brackets, some don't allow
- For those that don't allow, use special character classes inside
[]. Refer to [[linux:regex:character class][linux:regex:character class
- For those that don't allow, use special character classes inside
. a character execpt newline or line terminator \w a word character including _ [A-Za-z0-9_] \W a non-word character. same as [$\w] \d a digit \D a non-digit character \s whitespace § a non-space character \b word boundary \0 a NUL character \n a new line character \f a form feed character. e.g. page/section break \r a carriage return \t a tab character \v a vertical tab character \uxxxx a Unicode character \u0057 is w \xxx \127 is w \xdd \x57 is w - Some language can use metacharacters inside square brackets, some don't allow
- Non greedy match (?=n)
var s = 'Is this all there is'; var p = /is(?= all)/g; s.match(p); // is // (?= all) matches any string that is before ' all': but the matched is not returned. // The other way to intepret it is find any 'is' that has ' all' right behind var p = /is(?! all)/g; // Find any 'is' that doesn't have ' all' right behind
- Match HTML Tags regex:match_tags
Match all tags
'<[^>]*>'
Match one tag only
'<img[^>]*>' '<(img|/?div|br)[^>]*>'
8.6.10. Function
- A special type of object
- See js:hoisting
8.6.10.1. arguments
function add() { var sum = 0; for (var i = 0, j = arguments.length; i < j; i++) { sum += arguments[i]; } return sum; } add(2,3,4,5); // Use Spread Syntax js:spread function add(firstValue, ...args) { var sum = 0; for (let value of args) { sum += value; } sum += firstValue; return sum; } add(2,3,4,5); // Optional argument function verify(w) { w = (typeof w === 'undefined') ? 'default' : w; }
8.6.10.2. Default parameter
// Not supported: IE, Safari < 10 function multiply(a, b = 1) { return a * b; } // backward compatible function abc(a){ if(a === undefined){ a = 2; } }
8.6.10.3. .apply(), .call()
.apply()- Pass arguments as an array to a function which accepts multiple arguments
funct.apply(thisArg, [argsArray])- pass thisArg into funct as
thisin funct - array or array-like object
- Pass no more than 65536 arguments/elements in array
.call(thisArg, arg1, ... , argN)- Pass arguments as arguments to a function which accepts multiple arguments
thisArg- pass it into funct as
thisin funct - (no term)
- See IIFE
var numbers = [5,6,2,3,7]; var max = Math.max.apply(null, numbers); // Equivalent to Math.max(5,6, ...); var max = Math.max.call(null, 5,6,2,3,7);
They are also used to call a method on a varible (obj or array) which doesn't have that method as its property.
- Loop arrays which don't have .forEach() such as NodeList
8.6.10.4. .bind()
- Create a new function with
thismodified inside the new function. Not modify a function let boundFunc = func.bind(thisArg[, arg1[, arg2[, ...argN]]])- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Function/bind
var cb = function(rq) { this.successCB(rq); this.failCB(rq); }; cb = cb.bind({successCB: cbSuccess, failCB: cbFail}); aNormalFunctionRunsACallback(cb);
8.6.11. Exception, Error
try { throw "Too big"; // Manually throw an error var err = new Error("Too big"); throw err; } catch (err) { // err === 'Too big' if it's a string console.log( err.message, // Standard err.description, // Microsoft. err.number, // Microsoft. same as linenumber err.linenumber, // FF and Chrome err.name, // Standard. Initial is Error but you can change it err.stack // Not standard but works in all browsers.. Stack trace. ); } finally { // success or failure, run this }
8.6.12. Prototype Object js:prototype
- Every js:Object has a prototype object
- prototype object's default properties and methods
- constructor
- property. get/set
Object.getPrototypeOf(anObj)- e.g.
Object.getPrototypeOf(document.getElementById('abc')) - (no term)
Get all keys (not only enumerable) in an object including prototype-chain properties and non-enumerable properties
function logAllProperties(obj) { if (obj == null) return; // recursive approach console.log(Object.getOwnPropertyNames(obj)); logAllProperties(Object.getPrototypeOf(obj)); } logAllProperties(my_object);
- Compared to js:Object.keys which only gets own enumerable properties
- hasOwnProperty('aProperty')
bool
objA.hasOwnProperty('aProperty')- not
objA.prototype.hasOwnProperty('aProperty')!
aPropertyhas to be direct (notobjA.prototype.aProperty) and not inherited- Use
if ('key' in myObj)to check all enumerable keys js:for…in
- isPrototypeOf()
- js:prototype.isPrototypeOf
- Parent.prototype.isPrototypeOf(SubParent.prototype) is true
- Whether or not SubParent has some prototype properties removed, added or modified
- js:instanceof
- propertyIsEnumerable
- js:prototype.propertyIsEnumerable
- js:Enumerable
- objA.propertyIsEnumerable('prop') not objA.prototype.propertyIsEnumerable('prop')
- return true if it's enumerable in js:for…in
- inherited properties are not enumerable
- but newly created properties or methods are enumerable!
- js:inheritance
- Also, if any prototype property is modified in the upper chain, the lower chain and its instantiated objects will be modified, too!
8.6.13. Symbol js:symbol
- Since js:es6
- A symbol value is created by invoking function Symbol, which dynamically produces an anonymous, unique value
// here are two symbols with the same description, let Sym1 = Symbol("Sym"); let Sym2 = Symbol("Sym"); console.log(Sym1 == Sym2); // returns "false" // Symbols are guaranteed to be unique. // Even if we create many symbols with the same description, // they are different values. alert(Sym1); // TypeError: Cannot convert a Symbol value to a string alert(Sym1.toString()); // now it works alert(Sym1.description); // Sym const id = Symbol(); const courseInfo = { title: "ES6", topics: ["babel", "syntax", "functions", "classes"], id: "js-course" }; courseInfo[id] = 41284; console.log(courseInfo); // courseInfo has a unique key `Symbol()` which is non-representable, the value of that key is 41284
Symbol can be used to extract an object's iterator function
var title = 'ES6'; console.log(typeof title[Symbol.iterator]); var iterateIt = title[Symbol.iterator](); console.log(iterateIt.next()); console.log(iterateIt.next()); console.log(iterateIt.next()); console.log(iterateIt.next()); function tableReady(arr) { var nextIndex = 0; return { next() { if(nextIndex < arr.length) { return {value: arr.shift(), done: false} } else { return {done: true} } } } } var waitingList = ['Sarah', 'Heather', 'Anna', 'Meagan']; var iterateList = tableReady(waitingList); console.log(`${iterateList.next().value}, your table is ready`); console.log(`${iterateList.next().value}, your table is ready`); console.log(`${iterateList.next().value}, your table is ready`); console.log(`${iterateList.next().value}, your table is ready`); console.log(`Is this finished? ${iterateList.next().done}`);
8.6.14. Type Conversion
Javascript conversion is automatic. var y = "5"; var x = + y;
General methods: String(x) or x.toString() / convert to string Number(x) / convert to numbers
8.6.14.1. Number to String
x.toString(); x.toExponential(2); / 2 decimal points: 9.66e+0 x.toFixed(2); / 2 decimal points x.toPrecision(); // total number of digits
8.6.14.2. String to Number
parseInt("10"); / 10 parseInt("10.33"); / 10 parseInt("10.53"); / 10 parseInt("10 20 30"); / 10 parseInt("10 years"); / 10 parseInt("year 10"); / NaN
The same for parseFloat.
Always provide a base number :: parseInt("10", 10); // decimal base
8.7. Ternary
- See 1.14
// Ternary var myVar = document.getElementById('myCheckbox'); myVar ? myVar.checked : null // But Elvis is not supported myVar ?: null // May consider chaining to see if deeper key is defined // undefined if either a or b are null/undefined, otherwise a.b.c (a has to be defined, otherwise error) const myVar = a && a.b && a.b.c; // eq. to the new Optional Chaining Operator `?.` (IE not supported) (a has to be defined, otherwise error) const myVar = a?.b?.c; // Nullish Coalescing operator `??` can also be used to set default value (IE not supported) (a has to be defined, otherwise error) const myVariable = a?.b?.c ?? 'Some other value'; // Evaluates to 'Some other value' const myVariable2 = null ?? 'Some other value'; // Evaluates to '' const myVariable3 = '' ?? 'Some other value'; // Exmaple about DOM var myCheckbox = document.getElementById('myCheckbox')?.checked; var postData = []; if (myCheckbox !== undefined) { postData.push('myCheckbox=' + (myCheckbox ? 1 : 0)); }
8.8. Loops and Breaks
8.8.1. for, while, do…while
var cars = ['a', 'b', 'c']; for (var i = 0, len = cars.length, text = ""; i< len; i++) { // Loop through all properties of Array.prototype // 3rd party libraries or frameworks might add methods or properties to Array.prototype // So this is not recommended for looping an Array } for (var i = 0; i < cars.length; i++) { } while () { } do { } while (condition); // Loop child elements, all children elements var investTypesDiv = document.getElementById('MA_lead_final_investment_type'); var investAllTypes = []; for (let i = 0; i < investTypesDiv.children.length; i++) { investAllTypes.push(investTypesDiv.children[i]); }
8.8.2. for…in js:for…in
- Loop through all enumerable properties of an object including prototype-chain properties
- Can be used to loop through arrays
- The retrieval order is not the same as the object define order! For both objects and arrays
var myObj = {}; for (var prop in myObj) { console.log(myObj[prop]); } let myArray = ["a","b","c"]; for (let key in myArray) { // key is a string! let value = myArray[key]; console.log("key: %o, value: %o", key, value); }
8.8.3. for…of js:for…of
- js:for…in interates over all enumerable properties of an object including prototype-chain properties
- js:for…of iterates over elements of any collection that has a
[Symbo.iterator]property- interates over iterable objects
Array, Map, Set, String, TypedArray, arguments but not objects!
- Use it with
Object.keys(myObj)to loop through object keys
const keys = Object.keys(fruits) for (const key of keys) { console.log(key) }
Object.keys(myObj)returns an array of enumerable own properties not including prototype-chain properties. Supported by all browsers- Compare to js:Object.getOwnPropertyNames which gets all keys!
Object.values(myObj)returns an array of valuesObject.entries(myObj)returns[ ["name","John"], ["age",30] ]- IE not supported
Example
const usersMap = { id_1: { name: 'John 1', }, id_2: { name: 'John 2', }, }; const allUsersWithId = Object.entries(usersMap); allUsersWithId.forEach(([userId, user]) => { console.log(userId, user); });
- Refer to js:prototype
- Works with NodeList
const elements = document.querySelectorAll('.foo'); for (const element of elements) { element.addEventListener('click', doSomething); }
- Use it with
- (no term)
- Order is the same as the definition!
- (no term)
- Since ES2015
for (let value of iterable) { console.log(value); } // loop Map or Set var topics = new Map(); topics.set('HTML', '/class/html'); topics.set('CSS', '/class/css'); topics.set('JavaScript', '/class/javascript'); topics.set('Node', '/class/node'); for (let topic of topics) { console.log(topic); // ['HTML', '/class/html'] } for (let topic of topics.keys()) { console.log(topic, "is the course name"); // 'HTML' } for (let topic of topics.values()) { console.log("The course description can be found at", topic); // '/class/html' } for (let course of topics.entries()) { console.log(course); // ['HTML', '/class/html'] }
8.8.4. Array.prototype.forEach(callback[, thisArg]) js:forEach
- Arguments
- thisArg
- Pass a variable outside of .forEach() and change it inside .forEach() using
this - callback
- 3 arguments
- currentValue
- index
- array
- Return
- undefined. So
.forEach()is not chainable
For objects that have no reference to
Array.prototypee.g.document.getElementsByClassName('myClass')[].forEach.call(elements, function(ele) { // Do something });
- Can't
breakcompared to traditional loop - Elements appended after the call to forEach() begins will not be visited
- The value of element is changed before forEach() visits, it will reflect the change
- Elements removed before being visited will not be visited
8.8.5. Array.prototype.every()
- Test whether all elements in the array pass the test implemented by the provided function. It returns a Boolean value.
- Can break but not using
break
// Inline callback function every(function (element) { /* … */ }) every(function (element, index) { /* … */ }) every(function (element, index, array) { /* … */ }) every(function (element, index, array) { /* … */ }, thisArg) let arr = [1, 2, 3, 4]; arr.every((elem, index, arr) => { arr[index + 1]--; console.log(`[${arr}][${index}] -> ${elem}`); return elem < 2; // if it is false, every() will return false and stop }); // 1st iteration: [1,1,3,4][0] -> 1 // 2nd iteration: [1,1,2,4][1] -> 1 // 3rd iteration: [1,1,2,3][2] -> 2
8.8.6. Array.prototype.some()
- Similar to
every(), but only one item/element must pass the test.
8.8.7. break and continue
break- without a label reference, can only jump out of a loop or a switch
- With a label reference,
breakcan jump out of any code block
- with or without a label, can only skip one loop iteration
A code block is a block of code between { and }
list: {
text += cars[0] + "<br>";
text += cars[1] + "<br>";
break list;
text += cars[3] + "<br>";
}
8.9. Enumerable js:Enumerable
Except inherited properties, all methods and properties are enumerable in js:for…in
- Refer to js:prototype.propertyIsEnumerable
- Refer to js:Object.defineProperty
8.10. JSONP, JSON
8.10.1. JSONP
<script type="text/javascript" src="http://b.com/returnjsondata?callback=mycallback"></script> Where b.com/returnjsondata returns a json object. And the whole <script> in yourwebsite.com will become mycallback({foo:'bar'});
8.10.2. JSON
- JSON object names require double quotes
- String are wrapped in double quotes
- Boolean is
trueorfalsein lowercase nullcan be used- MIME type: application/json
var obj = JSON.parse(jsonString);obj._embedded['wp:featuredmedia'][0]
8.10.2.1. Deep Keys
// see if test.level1.level2.level3 key exists var r1 = (((test || {}).level1 || {}).level2 || {}).level3; if (typeof r1 !== 'undefined') { }
8.11. DOM Manipulation
8.11.1. If an element exists
var e = document.getElementById("abc"); if (!!e) { // exists } var e = document.getElementsByClassName('aClass'); if (e.length > 0) { // exists }
8.11.2. Loop querySelectorAll
var divs = document.querySelectorAll('.aClass'); [].forEach.call(divs, function(div) { div.style.color="red"; }); // Or divs.forEach(function() { // });
8.11.3. Remove a class
classListreturns DOMTokenList object- Don't need to check existance
ELEMENT.classList.remove("CLASS_NAME"); ELEMENT.classList.add('hint'); e.classList.toggle('CLASS_NAME'); e.classList.contains('CLASS_NAME'); // bool
8.11.4. Selected Option
var e = document.getElementById("selectElementID"); var i = e.selectedIndex; var ep = e.options[i]; ep.value; e.value; // same as ep.value ep.text;
8.11.5. Meta tag
this.ncmGoogleTagManager = this.ncmGoogleTagManager || {}; var ns = this.ncmGoogleTagManager; ns.getMetaTagByProperty = function (property) { var metas = document.getElementsByTagName('meta'); var content = []; for (i = 0; i < metas.length; i++) { if (metas[i].getAttribute('property') == property) { content.push(metas[i].getAttribute("content")); } } return content; }; ns.ArticlePubTime = ns.getMetaTagByProperty('article:published_time'); if (ns.ArticlePubTime.length) { return ns.ArticlePubTime[0]; } else { return null; }
8.11.6. Element data attributes
<div id="id" data-primary="abc"></div> <link rel='shortlink' href='https://www.a.ca/?p=123' />
var e = document.getElementById('id'); if (e.dataset.hasOwnProperty('primary')) { print e.dataset.primary; e.dataset.primary = 'new primary'; } if ( i = document.querySelector("link[rel=shortlink]")) { console.log(i.getAttribute('href')); }
8.11.7. Remove all children
var e = document.getElementById('abc'); while (e.firstChild) { e.removeChild(e.firstChild); } var node = document.createElement('img'); node.src = 'https://a.ca/b.png'; e.appendChild(node);
8.12. URI and URI Component Encode, Current URL, open, email link
8.12.1. encodeURIComponent vs encodeURI
- eq. to php:urlencode
- See 8.23
- For URL parameters and POST data
var url1 = "http://abc.com"; var url2 = "http://xyz.com/test.php?url=" + encodeURIComponent(url1); var array1 = [1, 2, 3]; encodeURIComponent(JSON.stringify(array1)); // decodeURIComponent(); // When you want to encode a whole URL that you don't trust var uri = "http://abc.com/folder name with space/file name with space.asp?abc=has space" var uri_en = encodeURI(uri); // decodeURI(str) // encodeURI DO NOT encode these but encodeURIComponent does: @+/=&#$,:;? // Neither encodeURI nor encodeURIComponent encodes: '*_-.()~! // Both encodeURI and encodeURIComponent encode: %"{}<>Space
8.12.2. Current URL
console.log(window.location.href); // whole URL // in an iFrame, get the parent window url: window.parent.location // window.location.href === window.location.toString() === 'http://abc.com/xyz/ijk.php?m=123#def' // window.location.href = window.location.protocol + "//" // + window.location.hostname (abc.com) // + window.location.pathname (/xyz/ijk.php, if homepage, '/') // + window.location.search (?m=123) // + window.location.hash; (#def) // click to email function sendMail() { var link = "mailto:me@example.com" + "?cc=myCCaddress@example.com" + "&subject=" + encodeURIComponent("This is my subject") + "&body=" + encodeURIComponent(document.getElementById('myText').value); window.location.href = link; // redirect } // open a new tab window.open('/your-url', '_blank'); // Get URL Parameter var getUrlParameter = function getUrlParameter(sParam) { var sPageURL = decodeURIComponent(window.location.search.substring(1)), sURLVariables = sPageURL.split('&'), sParameterName, i; for (i = 0; i < sURLVariables.length; i++) { sParameterName = sURLVariables[i].split('='); if (sParameterName[0] === sParam) { return sParameterName[1] === undefined ? true : sParameterName[1]; } } }; // Given http://dummy.com/?technology=jquery&blog=jquerybyexample var tech = getUrlParameter('technology'); // Get URL Parameter from hash const queryString = window.location.hash.substr(1); let urlParams = new URLSearchParams(queryString); // not supported in IE let newToken = urlParams.get('access_token'); // Get last path var path = window.location.pathname; var lastindex = path.lastIndexOf('/'); if ( lastindex > 0 && path.length === lastindex + 1) { // remove last slash path = path.substring(0, path.length - 1); lastindex = path.lastIndexOf('/'); } console.log(path.substring(lastindex + 1)); // Remove hash function removeHash() { history.pushState("", document.title, window.location.pathname + window.location.search); }
8.13. Cookie js:cookie
https://developer.mozilla.org/en-US/docs/Web/API/Document/cookie
cookieVal = "abc"; cookieKeyValPair = "litest=" + encodeURIComponent(cookieVal); // only one single cookie can be set/updated at a time document.cookie = cookieKeyValPair; // Optional cookie attribute values, preceded by a semi-colon separator var d = new Date(); d.setTime(d.getTime() + 30*24*60*60*1000); // 30 days. If not set, cookie will be exired after browser is closed. var expires = "expires=" + d.toUTCString(); cookieKeyValPair += "; " + expires; cookieKeyValPair += "; path='/mydir'"; // or cookieKeyValPair += "; path=/" function setCookie(cname, cvalue, exdays) { var d = new Date(); d.setTime(d.getTime() + (exdays*24*60*60*1000)); var expires = "expires="+ d.toUTCString(); document.cookie = cname + "=" + encodeURIComponent(cvalue) + ";" + expires + ";path=/"; }
8.14. Object only, no Class js:Object
8.14.1. Define an object
var key = 'hello'; var a = {key: 123}; // {key: 123} var b = {}; b[key] = 123; // b === {hello: 123} var c = { [key]: 123 }; // ES6 c === {hello: 123}
8.14.2. Contructor new, Object.create
- Every thing in JavaScript is an object. No class
- Every object has a prototype object
- Example
Parentis an object with a constructor (because it's a function) and a prototype object- When
newkeyword is used,var parent = new Parent();,parentcopies theParent's prototype object and run the Parent's constructor - If the constructor returns anything, parent will take it
- If the constructor returns nothing, the
Parentobject with prototype object andParent's other properties defined in the constructorthis.xyz = ...will be copied - copies Parent's prototype object and return it. So
Parent's constructor is not run js:prototype
var a = Object.create(null);Create an object with extra properties
Object.create(Object.prototype, { foo: {writable: true, configuration: true, value: 'hello', get: ..., set: ..., otherthings: ...}, bar: {...} })
8.14.3. Methods of Object constructor
8.14.3.1. Object.create()
8.14.3.2. Object.freeze() js:Object:freeze
const obj = { prop: 42 }; Object.freeze(obj); obj.prop = 33; // Throws an error in strict mode console.log(obj.prop); // expected output: 42
8.14.4. instanceof js:instanceof
- A useful operand that is for object to test instantiation
myobject instanceof Myconstructorobject- an instance (an object)
Myconstructor- Function object to test against. It has to be a function name
- (no term)
- Return true if
Myconstructor.prototypeexists inmyobject's prototype chain
- Use
instanceofto test instantiation, and use js:prototype.isPrototypeOf to test inheritance
8.14.5. Object.defineProperty js:Object.defineProperty
// To make a method non enumerable // - `CustomerBooking` class and its method `setFilm` is already defined Object.defineProperty(Customerbooking.prototype,'setFilm', {enumerable:false}); // Define a new non-enumerable key `privateField` and value `5` Object.defineProperty(obj, 'privateField', {value: 5});
8.14.6. object.assgin Object assignment
var anyVariable = anyObject; // pass by reference const food = {beef: 'beef', bacon: 'bacon'} //region Non-Deep clone an object // Only enumerable properties are copied :: https://www.samanthaming.com/tidbits/70-3-ways-to-clone-objects/ // "Object.assign" // mutates target object food2 so that food2 copies all enumerable own properties from 1 or more source objects. // It also returns the target object var food2 = {}; console.log(Object.assign(food2, food)); // This is way is preferred var food3 = Object.assign({}, food) // JSON, functions are replaced with null and functions in object are replaced with undefined var food2 = JSON.parse(JSON.stringify(food)); // Spread // Shallow copy :: the first level is copied but deeper levels are referenced var food2 = {...food} //endregion
8.14.7. Compare objects without methods (JSON-style)
JSON.stringify(obj1) === JSON.stringify(obj2)
8.14.8. Reuse previous value when declaring
var obj = { key1: "it ", key2: "", get key3() { console.log("key3 get is run multiple times"); return this.key2 + " hello"; }, set init(dummy) { console.log("key2 is set only once"); this.key2 = this.key1 + " hello"; } }; obj.init = ""; console.log(obj.key2 === obj.key1 + " hello"); console.log(obj.key3 === obj.key2 + " hello"); console.log(obj.key3 === obj.key2 + " hello"); obj.key3 = "override"; console.log(obj.key3 === obj.key2 + " hello");
8.14.9. Example
function CustomerBooking(bookingId, customerName, film, showDate) { /* This is a constructor. It is not a good way to define property * Better to use methods to set. But here should provide initialization */ this.customerName = customerName; this.bookingId = bookingId; this.showDate = showDate; this.film = film; // You can also call a method which is defined later to initialize properties. } CustomerBooking.prototype.getCustomerName = function() { return this.customerName; } CustomerBooking.prototype.setFilm = function(film) { this.film = film; } // Instantiate var firstBooking = new CustomerBooking(1234, "Arnold Palmer", "Toy Story", "27 July 2004 20:15"); var enumerated = []; for (var prop in firstBooking) { enumerated[enumerated.length] = prop; } // ["customerName","bookingId", "showDate", "film", "getCustomerName", ..., "setFilm"] // A cinema can hold multiple CustomerBooking instances function cinema() { this.bookings = new Array(); } cinema.prototype.addBooking = function(bookingId, customerName, film, showDate) { this.bookings[bookingId] = new CustomerBooking(bookingId, customerName, film, showDate); } // Loop through bookings of a cinema cinema.prototype.getBookingsTable = function() { var booking; var bookingsTableHTML = "<table border=1>"; for (booking in this.bookings) { bookingsTable += this.bookings[booking].getBookdingId(); ... } bookingsTableHTML += "</table>"; return bookingsTableHTML; } var londonOdeon = new cinema(); // Constructor takes no parameters londonOdeon.addBooking(342, "First Last", "Toy Story", "15 July 2004 20:15");
8.15. Extending Native Object
Native prototype can't be deleted or replaced But values of its properties can be modified or created
Create a new method in Array class which removes a member after checking if the method is already defined
Array.prototype.remove = Array.prototype.remove || function(member) { var i = this.indexOf(member); if (i > -1) { this.splice(index,1); } return this; } // If a method (remove) is created in native class, but later it was assigned as a property error will show
Newly added remove method is enumerable in js:for…in js:prototype.propertyIsEnumerable Inherited properties are not enumerable To filter out method in js:for…in
var props = []; for (var prop in results) { results.hasOwnProperty(prop) && props.push(prop); }
Add "method" in Function class Then "method" exists in any function.
Function.prototype.method = function(name, func) { this.prototype[name] = func; return this; } // Parenizor is a custom class. Create a method called 'setValue' Parenizor.method('setValue', function(v) { this.value = value; return this; });
Add "inherits" in Function class
Function.method('inherits', function(parent) { // parent is a class that is already defined // make a new instance this.prototype = new parent(); var d = {}, p = this.prototype; this.prototype.constructor = parent; this.method('uber', function uber(name) { if (!(name in d)) { d[name] = 0; } var f, r, t = d[name], v = parent.prototype; if (t) { while (t) { v = v .contructor.prototype; t -= 1; } f= v[name]; } else { f = p[name]; if (f == this[name]) { f = v[name]; } } d[name] +=1; r = f.apply(this, Array.prototype.slice.apply(arguments, [1])); d[name] -=1; return r; }); return this; });
8.16. Classical Inheritance, Parasitic Inheritance js:inheritance
Classical inheritance is about the is-a relationship. Parasitic inheritance is about the was-a-but-now's-a relationship.
8.16.1. Classical inheritance
8.16.1.1. Example
function Parenizor(v) { // enclose v with ( and ) this.setValue(v); } Parenizor.prototype = { constructor: Parenizor, // This line is crucial as we are overwriting Parenizor.prototype rather than adding setValue: function(v) { this.value = v; return this; }, getValue: function() { return this.value; }, toString: function() { return '(' + this.getValue() + ')'; } }; myParenizor = new Parenizor(0); myString = myParenizor.toString(); // "(0)" // Subclass inherits parent class Parenizor function ZParenizor(v, w) { Parenizor.call(this, v); // call parent class parenizor constructor // You can also call other parent class constructor // OtherParenizor.call(this, v); // Define other properties that this subclass should have this.otherValue = w; } ZParenizor.prototype = Object.create(Parenizor.prototype); // Inherits parent class's all methods. // First create an instance of the parent class and assign it to the child class. // Inherits other parent class's all methods // mixin(ZParenizor.prototype, OtherParenizor.prototype); ZParenizor.prototype.constructor = ZParenizor; // Set subclass constructor // but ZParenizor should have its own constructor which is ZParenizor.prototype.constructor. // Modify subclass's inherited method toString ZParenizor.prototype.toString = function() { // ... } var zParenizor = new ZParenizor('v', 'w'); console.log(zParenizor instanceof ZParenizor); // true console.log(zParenizor instanceof Parenizor); // true
8.17. OOP Pattern
- This is not a singleton pattern. See 8.5.7
// Simple OOP pattern or prototypal pattern Vehicle = function (year, make, model) { //region Public properties this.year = year; this.make = make; this.model = model; //endregion }; Vehicle.prototype.publicProperty = ''; Vehicle.prototype.getInfo = function () { // Use private properties return this.year + ' ' + this.make + ' ' + this.model; }; Vehicle.prototype.startEngine = function () { return 'Vroom'; }; (function () { // this === self === window this.myApp = this.MyApp || {}; var ns = this.myApp; // further shorten the keyboard typing // private properties var vehicleCount = 5; var vehicles = []; // Public properties ns.publicHello = 'This is public'; //region Public methods ns.getVehicleCount = function () { return vehicleCount; }; ns.setVehicleCount = function (value) { vehicleCount = value; }; //endregion // Define a Class // You can even separate the class definition into another JavaScript file // Just remember to include the necessary closure like the above ns.Vehicle = (function () { // Don't Do This! // - Since it's not in prototype, this privateStaticProperty cannot be inherited // - And if any prototypal method uses this variable in child class, it will throw error var privateStaticProperty = 'Vehicle Private Property'; function Vehicle(year, make, model) { //region public properties this.year = year; this.make = make; this.model = model; //endregion } //region Public methods Vehicle.prototype.getInfo = function () { // Use public properties return this.year + ' ' + this.make + ' ' + this.model; }; Vehicle.prototype.startEngine = function () { return 'Vroom'; }; Vehicle.prototype.setPrivateStaticProperty = function (value) { privateProperty = value; }; Vehicle.prototype.getPrivateStaticProperty = function () { return privateProperty; }; //endregion // Better not to override prototype, don't do this: // Vehicle.prototype = {getInfo: function () {}, startEngine: function () {}} return Vehicle; }()); // Define a sub Class ns.Car = (function (parent) { function Car(year, make, model) { parent.call(this, year, make, model); this.wheelQuantity = 4; } // Inherit parent's prototype properties Car.prototype = Object.create(parent.prototype); // Override the constructor to this child class Car.prototype.constructor = Car; // Extend or override a parent's method Car.prototype.getInfo = function () { return 'Vehicle Type: Car ' + parent.prototype.getInfo.call(this); } return Car; }(ns.Vehicle)); }()); console.log(myApp.publicHello); // undefined for private variable myApp.vehicleCount myApp.getVehicleCount(); myApp.setVehicleCount(10); // Instantiate var v = new myApp.Vehicle(2012, 'Toyota', 'Rav4'); console.log(v.getInfo()); var c = new myApp.Car(2012, 'Toyota', 'Rav4'); console.log(c.getInfo()); // Instantiate Simple OOP var vSimple = new Vehicle(2022, 'Lexus', 'UX'); console.log(vSimple.getInfo());
8.18. Promise
- Promise is now native in all browsers including Edge except IE
IE polyfill https://github.com/stefanpenner/es6-promise
<script src="https://cdn.jsdelivr.net/npm/es6-promise@4/dist/es6-promise.auto.min.js"></script>
-
function get(url) { // You should wrap all code into return return new Promise(function (resolve, reject) { // run resolve or reject function to resolve or reject a promise var req = new XMLHttpRequest(); req.open('GET', url); // it can be set to async req.onload = function () { if (req.status == 200) { resolve(req.response); } else { reject(Error(req.statusText)); } }; req.onerror = function () { reject(Error("Network Error")); } }); } get('story.json') .then(function (response) { console.log('Success!', response); // transform a value to the next .then // just return the value, you may modify it // The returned value has to be a non-promise return JSON.parse(response); }, function (error) { console.log('Failed!', error); return error; }) .then(function (response) { console.log('JSON!', response); }); // If a 'then' just returns a value when it's resolved, use a shortform get('story.json').then(JSON.parse).then(function(response) { console.log('JSON!', response); }); // In 'then', you can return a promise // a promise.then is a promise function getJSON(url) { return get(url).then(JSON.parse); } getJSON('story.json').then(function(story) { return getJSON(story.chapterUrls[0]); }).then(function(chapter1) { console.log('Got Chapter 1', chapter1); }); // Store the result of a promise and reuse it var storyPromise; function getChapter(i) { storyPromise = storyPromise || getJSON('story.json'); return storyPromise.then(function(story) { return getJSON(story.chapterUrls[i]); }); } getChapter(0).then(function(chapter) { console.log(chapter); return getChapter(1); }).then(function(chapter) { console.log(chapter); }); // .catch(function(e) { ... }) equals to // .then(undefined, function(e) { ... }) // Dynamically chained .then() // Start with a resolved promise: Promise.resolve('initialValue') // a reject promise: Promise.reject('initialValue') getJSON('story.json') .then(function(story) { // addHTMLToPage(story.heading); return story.chapterUrls.reduce(function(sequence, chapterUrl) { return sequence .then(function() { return getJSON(chapterUrl); }) .then(function(chapter) { /* addHtmlToPage(chapter.html); */ }); }, Promise.resolve()); }) .then(function() { /* addTextToPage("All done"); */ }) .catch(function(err) { /* addTextToPage("broken" + err.message); */ }) .then(function() { /* document.querySelector('.spinner').style.display='none'; */ }); // Start a set of promises in parallel and create a promise that fulfils when all are fulfilled getJSON('story.json') .then(function(story) { // addHTMLToPage(story.heading); return Promise.all( story.chapterUrls.map(getJSON) ); // not a callback. It's an array of Promises (not chained) // After promises are added to array as values, they are started already // Start a set of promises in parallel, chain them so that we get response in order return story.chapterUrls.map(getJSON) .reduce(function (sequence, chapterPromise) { return sequence .then(function () { return chapterPromise; }) .then(function (chapter) { addHtmlToPage(chapter.html); }); }, Promise.resolve()); }) .then(function() { /* ... */ }) .catch(function(err) { /* ... */ }) .then(function() { /* ... */ });
8.19. Strict mode
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode
- Strict mode applies to entire scripts or to individual functions
Function
function myStrictFunction() { // Function-level strict mode syntax "use strict"; function nested() { return "And so am I!"; } return `Hi! I'm a strict mode function! ${nested()}`; }
- It doesn't apply to block statements enclosed in
{}braces; attempting to apply it to such contexts does nothing. evalcode,Functioncode, event handler attributes, strings passed toWindowTimers.setTimeout(), and related functions are entire scripts, and invoking strict mode in them works as expected
8.20. this and self
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/this
- It refers to an instance which is created using
newkeyword ora as object literals. Otherwise,thiswill default to thewindowscope - Arrow functions are different and follow the normal variable lookup rules
- So while searching for
thiswhich is not present in the current scope, an arrow function ends up finding thethisfrom its enclosing scope https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions
window.age = 10; function Person () { this.age = 42; // this Person instance (beause it is created by new Person) // will have property age as 42 console.log('this.age outside setTimeout', this.age); // 42 setTimeout(function () { // This is a traditional function (anonymous) and it is not created by // using `new` So `this` is the window scope console.log('this.age', this.age); // 10 }, 100); setTimeout(() => { // Arrow function executing in the "p" (an instance of Person) scope console.log('this.age', this.age); // yields "42" because the function // executes on the Person scope }, 100); } var p = new Person(); console.log('window.age', window.age); // 10
- So while searching for
- there is no self keyword in JavaScript. If JavaScript is run on a browser,
window.self = windowand all properties ofwindowcan be directly accessed. Soselfiswindowsunless it's overriden in other scopes
8.20.1. globalThis
- The global
thisvalue e.g. the global object - Different environments have different global
thisobject- On the web
window,self,frames- (no term)
- js:worker only uses
self - Node.js
global- Prior to
globalThis, the only reliable cross-platform way to get is Function('return this')()- Some environments that disable
eval()inFunction(''), like header:csp in browsers, prevent it this usage
- Some environments that disable
8.20.2. this in function context
function f1() { return this; } // In a browser: f1() === window; // true // In NodeJS: f1() === globalThis; // true function f2() { 'use strict'; // see strict mode return this; } f2() === undefined; // true
8.20.3. this in class context
- All non-static methods within the class are added to the prototype of
this- Static methods are properties of the class itself
class Example { constructor() { const proto = Object.getPrototypeOf(this); console.log(Object.getOwnPropertyNames(proto)); } first(){} second(){} static third(){} } new Example(); // ['constructor', 'first', 'second']
8.20.4. this in a method of an object
- The method is assigned into an instance (could be its original definition or other defintion)
thisvalue is the instance's value
8.20.4.1. Example 1
abc = {
a: 10,
b: function () {
return this.a;
},
};
// traditional function (anonymous) `b` is in an instance `abc`, the instance's `a` is 10
console.log(abc.b(), 'abc.b()'); // 10
console.log(window.a, 'window.a'); // undefined
var o = { prop: 37 };
function independent () {
return this.prop;
}
o.f = independent;
// the called traditional function `independent` is now in an instance `o`, the instance's `prop` is 37
console.log(o.f(), 'o.f()'); // 37
// the called traditional function `independent` is now in an instance `b`, the instance's `prop` is 42
o.b = { g: independent, prop: 42 };
console.log(o.b.g(), 'o.b.g()'); // 42
var o = { f: function () { return this.a + this.b; } };
var p = Object.create(o);
p.a = 1;
p.b = 4;
// the called traditional function `f` is now in an instance `p`, the instance's `a` is 1 and `b` is 4
console.log(p.f(), 'p.f()'); // 5
xyz = {
a: 100,
b: abc.b,
};
// abc.b now belongs to instance xyz and xyz's `a` is 100
console.log(xyz.b(), 'xyz.b()'); // 100
8.20.4.2. Example 2
function sum () { return this.a + this.b + this.c; } var o = { a: 1, b: 2, c: 3, get average () { return (this.a + this.b + this.c) / 3; }, }; Object.defineProperty(o, 'sum', { get: sum, enumerable: true, configurable: true, }); console.log(o.average, o.sum); // 2, 6
8.20.5. this in a DOM event handler
my_element.addEventListener('click', function (e) { console.log(this.className); // logs the className of my_element console.log(e.currentTarget === this); // always `true` console.log(this === e.target); // true when currentTarget and target are the // same object }); my_element.addEventListener('click', (e) => { console.log(this.className); // WARNING: `this` is not `my_element` console.log(e.currentTarget === this); // `false` });
In an inline event handler
<button onclick="alert(this.tagName.toLowerCase());"> <!-- alert `button` --> Show this </button> <button onclick="alert((function() { return this; })());"> <!-- alert the window object --> Show inner this </button> <script> function logID() { console.log(this.id); } </script> <table id="my_table" onclick="logID();"><!-- when called, `this` will refer to the global window object --> ... </table>
8.21. Observable js:observable
A promise is a push mechanism that calls some code after the promise has been resolved or rejected with a single value. An observable is like a promise, but it calls some code every time a new value becomes available, and can emit many values over time.
8.22. Worker
- IE 10 and above supports worker
- Workers are independent and separated from parent window.
- Don't try to access or modify parent window DOM. Do it indirectly
var w; if (typeof(Worker) !== "undefined") { if (typeof(w) == "undefined") { w = new Worker("demo_workers.js"); } // Receive message from worker w.onmessage = function(event) { console.log(event.data); } w.onerror = function(error) { console.log(error.message, error.filename, error.lineno); } // Send message to worker w.postMessage(['First Name', 'Last Name']); } w.terminate(); w = undefined; // demo_workers.js var i = 0; function timedCount() { i = i + 1; postMessage(i); setTimeout("timeCount()", 500); } close(); // Worker can close itself // This is how worker receives a message onmessage = function(e) { console.log("Message receved by worker from main window"); var workerResult = 'Result: ' + (e.data[0] * e.data[1]); console.log('Posting message back to main window'); postMessage(workerResult); } // Workers can spawn or create subworkers. URIs of subwokers are resolved relative to // the parent worker's location rather than that of the owning page. // Worker can load scripts within the same domain // Scripts are loaded and executed in order. importScripts('foo.js'); importScripts('foo.js','bar.js'); // Spawn a shared worker. IE doesn't support sharedWorker // Worker Square and Worker Multiply2Numbers both use shared worker Multiply // In sqaure.js and multiply2numbers.js if (!!window.SharedWorker) { var sharedWorker = new SharedWorker("multiply.js"); sharedWorker.port.postMessage([123,123]); // square.js sharedWorker.port.postMessage([123,789]); // multiply2numbers.js sharedWorker.port.start(); // Only needed in parent thread if onmessage is not defined // such as the shares worker is a callback of event listener in parent thread sharedWorker.port.onmessage = function(e) { // console.log(e.data); // receive from shared worker } } // In multiply.js shared worker port.start(); // only needed if onmessage is not defined in parent thread onconnect = function(e) { var port = e.ports[0]; port.onmessage = function(e) { var workerResult = 'Result: ' + (e.data[0] * e.data[1]); port.postMessage(workerResult); } }
8.23. XMLHttpRequest
8.23.1. Basics
- Event handlers
- onload
- Always use onload instead of onreadystatechange
- eq. to
onreadystatechangeandthis.readyState == 4
- eq. to
- (no term)
- onreadystatechange
- (no term)
upload
rq.upload.onprogress = function (e) { var gId = document.getElementById; gId('cancel').style.display = 'block'; if (e.lengthComputable) { gId('debug').innerHTML = Math.floor(e.loaded * 100 / e.total) + '%'; if (Math.floor(e.loaded * 100 / e.total) === 100) { gId('debug').innerHTML = 'Processing...'; } } else { gId('debug').innerHTML = 'uploading...'; } };
- (no term)
onabort
rq.onabort = function () { gid('cancel').style.display = 'none'; gid('debug').innerHTML = 'canceled'; }
- Others
- onloadstart, onprogress, onerror, ontimeout, onloadend
- Methods
- send
- setRequestHeader
- Specify request header
Content-typemanually to x-www-form-urlencoded andsend('postDataInStringWith&=Separated') send()with 8.23.3, Content-type is set tomultipart/formdataautomatically
- Specify request header
this.propertyName- readyState
- See 1.2.1
- (no term)
- responseText
- (no term)
- responseXML
- statusText
- "OK" or "Not Found"
function loadDoc() { var xhr = new XMLHttpRequest(); xhr.onload = function () { if (this.status === 200) { console.log( this.responseText // as text , this.responseXML // as it's XML data , this.statusText // "OK" or "Not Found" ); console.log(xhr.getAllResponseHeaders()); xhr.getResponseHeader("Last-Modified"); } }; xhr.onreadystatechange = function () { // if (xhr.readyState == 4 && xhr.status == 200) { ... } } xhr.open('GET', 'ajax_info.txt' , true // async , "username" // optional , "psw" // optional ); /* After open() */ xhr.setRequestHeader("Content-type", 'application/x-www-form-urlencoded'); // set request header. Especially if 'POST' form fields xhr.setRequestHeader('Accept', 'application/json, text/javascript'); // it's common for JavaScript client to add this request header xhr.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); // On PHP, the request header appears as $_SERVER['HTTP_X_REQUESTED_WITH'] // For receiving a binary string // xhr.overrideMimeType("text/plain; charset=x-user-defined"); /* before send(); */ xhr.send(); // Nothing when 'GET' /* For POST * var fname = encodeURIComponent('Henry'); * var varJson = encodeURIComponent(JSON.stringify(var1)); * xhr.send("fname="+fname+"&var2="+varJson); * if .setRequestHeader content-type is www-form-urlencoded * when data is received from PHP, you need to * json_decode(stripslashes($_POST['var2'])) */ // binary data can be sent /* var blob = new Blob(['abc123'], {type: 'text/plain'}); * xhr.send(blob); * var myArray = new ArrayBuffer(512); * var longInt8View = new Uint8Array(myArray); * for (var i=0; i< longInt8View.length; i++) { * longInt8View[i] = i % 255; * } * xhr.send(myArray); */ }
8.23.2. application/x-www-form-urlencoded vs multipart/form-data
- Common
- PHP will have
$_POSTif the request content-type is either of these 2 - To interpret other content type or other request method e.g. 5
$_POST = json_decode(file_get_contents('php://input'), true);$str = "first=value&arr[]=foo+bar&arr[]=baz";
- PHP will have
application/x-www-form-urlencoded- Transfers extra data if large amount of binary data is sent
- spaces are encoed as plus signs
multipart/form-data- HTML form submit uses js:formdata to POST
multipart/form-data Request header
Content-Type: multipart/form-data; boundary=user-agent-sets-this-boundaryuser-agent-sets-this-boundary
- can be any 1 to 70 characters and not ending with white space
- characters are a set of characters known to be very robust through email gateways. Use numbers only to be safe
- usually starts and ends with 2 dashes
Request Payload
--user-agent-sets-this-boundary Content-Disposition: form-data; name="gv-inv_image"; filename="a.jpg" Content-Type: image/jpeg --user-agent-sets-this-boundary Content-Disposition: form-data; name="description" some text --user-agent-sets-this-boundary--
- HTML form submit uses js:formdata to POST
8.23.3. FormData to send file
- Use JavaScript and HTML to POST
multipart/form-data - Not for IE 9 and below
HTML to include a file upload field
<form action="" method="post" enctype="multipart/form-data"> <div>Pictures: <input type="file" name="pictures[]" /> <input type="file" name="pictures[]" /> <input type="file" name="pictures[]" /> <input type="submit" value="Send" /> </div> <!-- Or just one --> <input type="file" id="gv-inv_image"> </form> <button onclick="upload(this);return false;">Upload</button>
var fileInputElement = document.getElementById('gv-inv_image'); var formData = new FormData(); formData.append("username", "Groucho"); formData.append("accountnum", 123456); // number 123456 is immediately converted to a string "123456" // if value is not string or Blob or file, it will be converted to string // HTML file input, chosen by user formData.append("userfile", fileInputElement.files[0]); // you can check the file before it gets uploaded // JavaScript file-like object var content = '<a id="a"><b id="b">hey!</b></a>'; // the body of the new file... var blob = new Blob([content], { type: "text/xml"}); formData.append("webmasterfile", blob); var request = new XMLHttpRequest(); request.open("POST", "http://foo.com/submitform.php"); request.send(formData); // request content type is multipart/form-data
php
if (isset($_FILES['gv-inv_image'])) { $file = $_FILES['gv-inv_image']['name']; $ext = pathinfo($file, PATHINFO_EXTENSION); $filename = sha1(rand(25, 1920) . rand(72, 2560)) . '-' . time(); $filename = strtoupper($filename) . '.' . $ext; if (move_uploaded_file($_FILES['gv-inv_image']['tmp_name'], '../uploads/' . $filename)) { $result = ['status' => 'OK', 'filename' => $filename]; } else { $result = ['status' => 'ERR', 'message' => 'File Move Error.']; } }
8.23.4. Receive binary as arraybuffer or blob
var oReq = new XMLHttpRequest(); oReq.open("GET", "/myfile.png", true); oReq.responseType = "arraybuffer"; oReq.onload = funciton(event) { var arrayBuffer = oReq.response; // not .responseText if (arrayBuffer) { // Read arraybuffer var byteArray = new Uint8Array(arrayBuffer); for (var i=0; i < byteArra.byteLength; i++) { // do something } // Read arraybuffer. // Construct a blob from arraybuffer var blob = new Blob( [arrayBuffer], {type: "image/png"} ); } }; oReq.send();
Specify the received data is blob
var oReq = new XMLHttpRequest(); oReg.open("GET", "/myfile.png", true); oReq.responseType = "blob"; oReq.onload = function(e) { var blob = oReq.response; } oReq.send();
8.23.5. Download a JSON file
index.php
function hello () { var xhr = new XMLHttpRequest() xhr.onload = function () { if (this.status === 200) { var disposition = this.getResponseHeader('content-disposition') var matches = /"([^"]*)"/.exec(disposition) // if the response file does not have a name, give a default file name file.json var filename = (matches != null && matches[1] ? matches[1] : 'file.json') var blob = new Blob([this.response], { type: 'application/json' }) var link = document.createElement('a') link.href = window.URL.createObjectURL(blob) link.download = filename document.body.appendChild(link) link.click() document.body.removeChild(link) } } // I found setting the response type is optional //xhr.responseType="blob"; xhr.open('GET', 'index2.php', true) xhr.send() }
index2.php
<?php http_response_code(200); header('Content-Disposition: attachment; filename="filename.json"'); $result = [ 'data' => 'hello' ]; echo json_encode($result);
8.24. File handling
8.25. Apply XSLT on XML
function loadXMLDoc(filename) { var xhttp = new XMLHttpRequest(); xhttp.open("GET", filename, false); try { xhttp.responseType = "msxml-document"; } catch(err) {} xhttp.send(); return xhttp.responseXML; } var xml = loadXMLDoc('cdcatalog.xml'); var xsl = loadXMLDoc('cdcatalog.xsl'); var xsltProcessor = new XSLTProcessor(); xsltProcesser.importStylesheet(xsl); resultDocument = xsltProcesser.transformToFragment(xml, document); document.getElementById("example").appendChild(resultDocument);
8.26. Apply XPath
var x=loadXMLDoc("books.xml");
var xml = x.responseXML;
path = "/bookstore/book[1]/title";
var nodes = xml.evaluate(path, xml,
null, // namespaceResolver
XPathResult.ANY_TYPE, // resultType
null // If an existing XPathResult object is specified, it will be reused
); // Return xpathResult object
var result = nodes.iterateNext();
while (result) {
console.log(result.childNodes[0].nodeValue);
result = nodes.iterateNext();
}
8.27. Web APIs
8.27.1. WebSocket js:ws
Javascript on client (e.g. web browser)
var connection = new WebSocket('ws://abc.com/echo', ['soap', 'xmpp']); // ['soap', 'xmpp'] are subprotocols // When connection is open, send some data connection.onopen = function() { connection.send('Ping'); // send string }; // When connection close connection.onclose = function() { // do something }; // Send ArrayBuffer as binary var img = canvas_context.getImageData(0, 0,400, 320); var binary = new Unit8Array(img.data.length); for (var i = 0; i < img.data.length; i++) { binary[i] = img.data[i]; } connection.send(binary.buffer); // Send file as Blob var file = document.querySelector('input[type="file"]').files[0]; connection.send(file); connection.onerror = function(error) { console.log('WebSocket Error ' + error); }; connection.onmessage = function(e) { console.log(e.data); }; // Define binary type before onmessage to receive binary connection.binaryType = 'arraybuffer'; // or blob connection.onmessage = function(e) { console.log(e.data.byteLength); };
8.27.2. Mutation Observer
MDN MutationObserver MDN MutationRecord
MutationObserver = window.MutationObserver; // callback function has 2 arguments // mutations: an array of objects of type MutationRecord // observer: this MutationObserver instance var observer = new MutationObserver(function (mutations, observer) { mutations.forEach(function (mutation) { // A MutationRecord has properties // Some of them might be different based on MutationObserver's options // Loop addedNodes // addedNodes return NodeList which doesn't have .forEach // js:function:call Array.prototype.forEach.call( mutation.addedNodes, function (addedNode) { if (addedNode.id == "bar") { console.log("bar was added!"); } }); }); }); // Define the parent element var target = document.getElementById('foo'); // Or $('#foo').get(0) // Which event should be observed var config = { attributes: true, attributeOldValue: true, // set to true if attributes is set to true // attributeFilter: ['attr1', 'attr2'], // filter out an array of attribute local names in observation childList: true, // add/remove child elements including text nodes subtree: true, // any change to the subtree including all descendants characterData: true, // maybe image or multimedia changes? characterDataOldValue: true }; observer.observe(target, config); // You may stop observing later observer.disconnect();
8.27.3. Intersection Observer js:intersection observer
Supported in Edge, Firefox, Chrome, Chrome Android but not Safar and iOS
It registers a callback function which is only executed when an element enters or exists another element or intersects at a threshold and thus saves event for every scroll or viewport movement.
It doesn't tell the exact number of pixels that overlap or specifically which ones they are; however, it convers the much more common use case of "If they intersect by somewhere around N%, I need to do something".
var io = new IntersectionObserver(
entries => {
console.log(entries);
},
{
// Using default options: callback will be called only when the element comes partially into view and when it completely leaves the viewport. Details below
}
);
// Start observing an element
io.observe(element);
// Stop observing an element
// io.unobserve(element);
// Disable entire IntersectionObserver
// io.disconnect();
Parameter entries are readonly IntersectionObserverEntry and they are delivered asynchronously and callback will run in the main thread.
- rootBounds
- DOMRect, the result of calling element.getBoundingClientRect() on the root element, which is the viewport by default.
- boundingClientRect
- DOMRect, the result of element.getBoundingClientRect() on the observed element.
- intersectionRect
- DOMRect, the intersection of the above 2 rectangles and tells which part of the observed element is visible
- intersectionRatio
- how much of the element is visible.
Options
- root
- Default
null - threshold
- Default
[0], callback is called when intersectionRatio is greater or lower (2 times), you can set [0, 0.25, 0.5, 0.75, 1] - rootMargin
- Default
"0px", grow or shrink the area used for intersections.
If an iframe observes one of its elements, both scrolling the iframe as well as scrolling the window containing the iframe will trigger the callback. For the latter case, rootBounds will be set to null to avoid leaking data across origins.
8.27.5. Beacon API
- https://golb.hplar.ch/2018/09/beacon-api.html
- Beacon request is POST request which does not require response
8.27.6. Fetch API
- https://caniuse.com/fetch
- No IE
- (no term)
CORS
fetch('https://example.com', { mode: 'cors', credentials: 'include' // send cookies })
- credentials
- optional. d
- (no term)
No timeout built-in
function fetchTimeout(url, init, timeout = 3000) { return new Promise((resolve, reject) => { fetch(url, init) .then(resolve) .catch(reject); setTimeout(reject, timeout); }) } // Or Promise .race([ fetch('http://url', {method: 'GET'}), new Promise(resolve => setTimeout(resolve, 3000)) ]) .then(response => console.log(response))
- (no term)
Force to abort. Have to use AbortController API
const controller = new AbortController(); fetch( 'http://domain/service', { method: 'GET' signal: controller.signal }) .then( response => response.json() ) .then( json => console.log(json) ) .catch( error => console.error('Error:', error) );
- (no term)
- No progress e.g. 30% download complete
- (no term)
- Cookies are not sent
8.28. Testing
8.28.1. Unit Testing
QUnit
8.28.2. Performance
var t0 = performance.now(); doSomething(); var t1 = performance.now(); console.log("Call to doSomething took " + (t1 - t0) + " milliseconds."); // Or this way for console only console.time('doSomething'); doSomething(); console.timeEnd('doSomething'); // On console, `console.timeEnd` prints out one line: // doSomething: 3.259765625 ms
8.29. Worker
8.29.1. Web Worker
- Workers are in external files which don't have access to DOM (objects like window, document and parent) and completely separate from the webpage
- Use a Web Worker for heavy computation on the client. Use a Service Worker for offline capabilities like intercepting network requests, as well as other fun stuff like push notifications and synchronization of content in the background.
- Web workers have many instances per tab while Service Worker has one for all tabs
- Web workers are terminated when the tab is closed
- Service worker can still receive messages when the tabs are closed but browsers are up
- In Android, service workers can still receive messages when browser apps are closed
- Service worker can still receive messages when the tabs are closed but browsers are up
- Service workers are also able to
postMessage - Both are completely async with main thread
// Place to create a web worker main.js var w; if (typeof(Worker) !== "undefined") { if (typeof(w) == "undefined") { w = new Worker("demo_workers.js"); } w.onmessage = function(e) { // handle message sent from web worker console.log(e.data); } // post message to web worker w.postMessage('hi'); // manually terminate a web worker w.terminate(); } else { // Worker not supported } // demo_workers.js var i=0; function timedCount() { i= i +1; postMessage(i); // post message from worker to the birth place setTimeout("timedCount()", 500); } timedCount(); onmessage = function(oEvent) { // receive messages sent from the birth place postMessage('Hi ' + oEvent.data); };
8.29.2. Service Worker js:serviceworker
8.29.2.1. Basics
- https://developers.google.com/web/fundamentals/primers/service-workers/
- https://developers.google.com/web/ilt/pwa/introduction-to-service-worker
- https://jakearchibald.github.io/isserviceworkerready/ IE11 is not supported
- HTTPS or localhost
- sw can/is
- cache files so that the website can work offline
- JavaScript Web Worker
- ES6
- 100% based on Promise
- Notifications API and Push API
- Background Sync API
- Channel Messaging API
- geofencing
- sw can't
- use localStorage
- access DOM directly
- Life Cycle
Registration
if ('serviceWorker' in navigator) { window.addEventListener('load', function() { navigator.serviceWorker.register('/sw.js').then(function(registration) { // Registration was successful console.log('ServiceWorker registration successful with scope: ', registration.scope); }, function(err) { // registration failed :( console.log('ServiceWorker registration failed: ', err); }); }); }
register()can be called many times without concern- Path can be relative path e.g. './sw.js', for event fetch, urls can also be relative
Say project is
/train/*and/train/sw.js, specify URLs to resource with different query string in mindregister('./sw.js') urlsToCache = [ './', // /train/ 'index.html', // /train/index.html 'images/a.jpg', // /train/images/a.jpg 'scripts/a.js' ]
- sw receives fetch events for everything on this domain
- sw only sees and receives fetch events for pages whose URL starts with
/ex/,/ex/page1/,/ex/page2/etcchrome://inspect/#service-workers- shows an sw exists https://yoursite.com/serviceworker with a pid which you can inspect and terminiate. Incognito kills all sw if Incognitor is closed
Installation In sw.js
var CACHE_NAME = 'my-site-cache-v1'; var urlsToCache = [ '/', '/styles/main.css', '/script/main.js' ]; self.addEventListener('install', function(event) { // Perform install steps event.waitUntil( caches.open(CACHE_NAME) .then(function(cache) { console.log('Opened cache'); // load other resources e.g. big resources that are not absolutely needed in first load // sw may be killed and unfinished downloading in this part may be dropped. // cache.addAll(...); // all resources should be downloaded otherwise this error throws and the install process fails // Uncaught (in promise) TypeError: Request failed return cache.addAll(urlsToCache); // Accept all URLs to fetch from and add response to the cache // if any files fail to download, the install step will fail // When all fetch promises are resolved, event activate event is triggered. }) ); return self.clients.claim(); // activate sw faster });
// when user navigates to a different page or refreshes, sw receives fetch events // return cache as response if cache has it, otherwise make a network request to fetch self.addEventListener('fetch', function(event) { event.respondWith( // find any cached results from any of the caches sw created. caches.match(event.request) .then(function(response) { // Cache hit - return response (cache value) if (response) { return response; } // fetch over the network return fetch(event.request); } ) ); });
- Activation
event.waitUntiladds 1 promise to theinstallandactivateevent callbacks. Multipleevent.waitUntilis possible inside the event callback and all promises have to settle- Event callbacks for
installandactivatehave to be finished before event fetch can be run return if cache is found, otherwise make a network request to fetch, check response, if not successful or not coming from the same origin, just return the response, otherwise cache that response
self.addEventListener('fetch', function(event) { event.respondWith( caches.match(event.request) .then(function(response) { // Cache hit - return response if (response) { return response; } // IMPORTANT: Clone the request. A request is a stream and // can only be consumed once. Since we are consuming this // once by cache and once by the browser for fetch, we need // to clone the response. var fetchRequest = event.request.clone(); return fetch(fetchRequest).then( function(response) { // Check if we received a valid response // 'basic' means it's a request from the current origin if(!response || response.status !== 200 || response.type !== 'basic') { return response; } // IMPORTANT: Clone the response. A response is a stream // and because we want the browser to consume the response // as well as the cache consuming the response, we need // to clone it so we have two streams. var responseToCache = response.clone(); caches.open(CACHE_NAME) .then(function(cache) { cache.put(event.request, responseToCache); }); return response; } ); }) ); });
fetch: always serve cache first, but always update cache
self.addEventListener('fetch', function(event) { event.respondWith( caches.open('mysite-dynamic').then(function(cache) { return cache.match(event.request).then(function(response) { var fetchPromise = fetch(event.request).then(function(networkResponse) { cache.put(event.request, networkResponse.clone()); return networkResponse; }) return response || fetchPromise; }) }) ); });
- Update Service Worker (sw.js)
- Update your service worker JavaScript file. When the user navigates to your site, the browser tries to redownload the script file that defined the service worker in the background. If there is even a byte's difference in the service worker file compared to what it currently has, it considers it new
- Your new service worker will be started and the install event will be fired
- At this point the old service worker is still controlling the current pages so the new service worker will enter a waiting state
- When the currently open pages of your site are closed, the old service worker will be killed and the new service worker will take control
- Once your new service worker takes control, its activate event will be fired
activateevent is fired when the service worker starts up. Long event activate may block page loads. Keep it as lean as possible, only use it for things you couldn't do while the old version was activeCreate
'pages-cache-v1'and'blog-posts-cache-v1'and delete old'my-site-cache-v1'self.addEventListener('activate', function(event) { var cacheWhitelist = ['pages-cache-v1', 'blog-posts-cache-v1']; event.waitUntil( caches.keys().then(function(cacheNames) { return Promise.all( cacheNames.map(function(cacheName) { if (cacheWhitelist.indexOf(cacheName) === -1) { return caches.delete(cacheName); } }) ); }) ); });
- To manually refresh sw.js on DevTools, Application > Service Workers > skipWaiting, or on the top to enable Update on reload
8.29.2.3. Caching strategies
https://jakearchibald.com/2014/offline-cookbook/
Cache when an action is triggered. Caches API is available from regular pages not only sw.js
document.querySelector('.cache-article') .addEventListener('click', function(event) { event.preventDefault(); var id = this.dataset.articleId; caches.open('mysite-article-' + id).then(function(cache) { fetch('/get-article-urls?id=' + id).then(function(response) { // /get-article-urls returns a JSON-encoded array of // resource URLs that a given article depends on return response.json(); }).then(function(urls) { cache.addAll(urls); }); }); });
Serve a offline page if both cache and network are not available
self.addEventListener('fetch', function(event) { event.respondWith( // Try the cache caches.match(event.request).then(function(response) { // Fall back to network return response || fetch(event.request); }).catch(function() { // If both fail, show a generic fallback: return caches.match('/offline.html'); // However, in reality you'd have many different // fallbacks, depending on URL & headers. // Eg, a fallback silhouette image for avatars. }) ); });
Sample
self.addEventListener('fetch', function(event) { // Parse the URL: var requestURL = new URL(event.request.url); // Handle requests to a particular host specifically if (requestURL.hostname == 'api.example.com') { event.respondWith(/* some combination of patterns */); return; } // Routing for local URLs if (requestURL.origin == location.origin) { // Handle article URLs if (/^\/article\//.test(requestURL.pathname)) { event.respondWith(/* some other combination of patterns */); return; } if (/\.webp$/.test(requestURL.pathname)) { event.respondWith(/* some other combination of patterns */); return; } if (request.method == 'POST') { event.respondWith(/* some other combination of patterns */); return; } if (/cheese/.test(requestURL.pathname)) { event.respondWith( new Response("Flagrant cheese error", { status: 512 }) ); return; } } // A sensible default pattern event.respondWith( caches.match(event.request).then(function(response) { return response || fetch(event.request); }) ); });
8.29.2.4. Push (not yet available on Chrome)
Update cache before showing a notification
self.addEventListener('push', function(event) { if (event.data.text() == 'new-email') { event.waitUntil( caches.open('mysite-dynamic').then(function(cache) { return fetch('/inbox.json').then(function(response) { cache.put('/inbox.json', response.clone()); return response.json(); }); }).then(function(emails) { registration.showNotification("New email", { body: "From " + emails[0].from.name tag: "new-email" }); }) ); } }); self.addEventListener('notificationclick', function(event) { if (event.notification.tag == 'new-email') { // Assume that all of the resources needed to render // /inbox/ have previously been cached, e.g. as part // of the install handler. new WindowClient('/inbox/'); } });
8.29.2.5. Background sync (not yet available on Chrome)
self.addEventListener('sync', function(event) { if (event.id == 'update-leaderboard') { event.waitUntil( caches.open('mygame-dynamic').then(function(cache) { return cache.add('/leaderboard.json'); }) ); } });
8.30. iFrame
8.30.1. Communication Between Children Window and Parent Window
- A parent window can have multiple iframes and iframes are loaded from other domain
#+NAME Post message from iFrame to parent Window
// In iframe parent.postMessage('the message','*'); // or window.top.postMessage // In parent window function receiveMessage(event) { if (event.origin !== 'http://otherdomain.com') return; console.log(event.data); } window.addEventListener("message", receiveMessage, false);
#+NAME Post message from parent window to iFrame
// In parent window window.onload = function() { var receiver = document.getElementById("iFrameID").contentWindow; receiver.postMessage('Hello', 'http://iframe.src.domain'); // '*' for any domain of any receiver } // In iFrame window.onload = function() { function receiveMessage(e) { if (e.origin !== "http://parent.window.domain") return; alert(e.data); } window.addEventListener('message', receiveMessage); }
8.30.2. Dynamic iframe height
This code requires the iframe domain is same origin.
<iframe id="preview_iframe" style="width:100%" onload="resize_iframe(this)" srcdoc="<?php echo str_replace('"', '"', $html);?>" > </iframe> <script> function resize_iframe(iframe) { iframe.height= iframe.contentWindow.document.body.scrollHeight + 'px'; } </script>
8.30.3. Responsive iframe
Refer to bootstrap:responsive helpers
8.31. setTimeout recall
function runTimeout() { setTimeout(function() { if (window.googletag && googletag.apiReady) { // do something } else { //console.log('googletag is not ready'); runTimeout(); } }, 5000); }
8.32. Device Detection
var isSafari = /^((?!chrome|android).)*safari/i.test(navigator.userAgent); console.log("Your screen resolution is: " + screen.width + "x" + screen.height); console.log("Your screen resolution is: " + window.screen.width * window.devicePixelRatio + "x" + window.screen.height * window.devicePixelRatio);
8.33. CSS Media Viewport Size js:viewport size
var w = Math.max(window.innerWidth || 0, document.body.clientWidth)var h = Math.max(window.innerHeight || 0, document.body.clientHeight)- width including scrollbars
- width excluding scrollbars
8.34. ECMAScript
- https://tc39.es/ecma262/
- 1999 or 2011. Fully supported since Chrome 41
- 6 stands for 6th edition. 2015 is the year js:es6
ES7-2016,ES8-2017,ES9-2018,ES10-2019
8.34.1. Arrow function
(a) => {
return a + 100;
}
(a) => a + 100;
a => a + 100;
a => {
return a + 100;
}
(a, b) => {
return a + b;
}
(a, b) => a + b;
(a, b, ...r) => expression
(a = 400, b = 20, c) => expression
([a, b] = [10, 20]) => expression
({ a, b } = { a: 10, b: 20 }) => expression
async param => expression
async (param1, param2, ...paramN) => {
statements
}
8.34.1.1. Double Arrow Functions (Currying)
// A simple arrow function const myFunction = (x, y) => { console.log('param1', x); console.log('param2', y); } // The same function in curried form or as you say, double arrow function const myFunction = x => y => { console.log('param1', x); console.log('param2', y); } // The above function without using arrow functions - function myFunction(x) { return function innerFunction(y) { console.log('param1', x); console.log('param2', y); } }
8.34.2. Map - New Data Type (ES6)
var course = new Map(); course.set('react', {description: 'ui'}); course.set('jest', {description: 'testing'}); console.log(course); // course.size === course.length === 2 console.log(course.react); // error console.log(course.get('react')); // an object {description: 'ui'} // details is equivalent to course var details = new Map([ ['react', {description: 'ui'}], // key/value can be any data type: [new Date(), 'today'], ['jest', {description: 'testing'}]// key/value can be any data type: ['items', [1, 2]] ]); console.log(details.size); details.forEach(function(item) { console.log(item); });
8.34.3. Set - New Data Type (ES6)
var books = new Set(); books.add('Pride and Prejudice'); books.add('War and Peace') .add('Oliver Twist'); console.log(books); console.log('how many books?', books.size); console.log('has Oliver Twist?', books.has('Oliver Twist')); books.delete('Oliver Twist'); console.log('has Oliver Twist still?', books.has('Oliver Twist')); var data = [4,2,4,4,2,5,1,6,7,5,6,8,2,7]; var set = new Set(data); // auto dedup console.log('data.length', data.length); console.log('set.size', set.size);
8.34.4. Generator Function
Treat each yield is a stop, and later code could be started when
gen.next()is calledfunction* eachItem(arr) { for(var i=0; i< arr.length; i++) { yield arr[i]; } } var letters = eachItem(["a", "b", "c", "d", "e", "f", "g"]); var abcs = setInterval(function(){ var letter = letters.next(); if(letter.done) { clearInterval(abcs); console.log("Now I know my ABC's"); } else { console.log(letter.value); } }, 500);
- Assign variable = yield (async requests) and program like they are normal sync. Refer to last example
Refer to nodejs:co for yielding promises
function* idMaker() { var index = 0; while (index <3) yield index++; } var gen = idMaker(); // Generator function is not contructable // don't use var gen = new idMaker; // When GF is called, an iterator object is returned and // the body is not executed yet. console.log(gen.next().value); // when the iterator's next() is called for the first time, // the GF body is executed until the 1st yield expression // when it's called the nth time, the nth yield expression is called. // Say there are 3 yields in total, when gen.next() is called the 4th time or more // it will be {value: undefined, done: true} // Use GF in another GF function* anotherGenerator(i) { var index = 0; while (index < 3) { index++; yield i + index; } } function* generator(i) { yield i; yield* anotherGenerator(i); yield i + 10; } var gen2 = generator(10); console.log(gen2.next().value); // ... // 10, 11, 12, 13, 20, done
Passing variable in next(). The variable passed will subsitute the whole yield statement for the previous iteration. (after the previous yield, and before the next/current yield)
function* consumer(){ while (true){ var val = yield null; console.log('Got value', val); } } var c = consumer(); c.next(1) // No 'Got value' c.next(2) // Got value 2
Usage Without GF, you will write
fs.readFile('blog_post_template.html', function(err, tpContent){ fs.readFile('my_blog_post.md', function(err, mdContent){ console.log(tpContent, mdContent); }); });
With GF
function readFile(filepath) { return function(callback) { fs.readFile(filepath, callback); } } function run(genfun) { var gen = genfun(); function next(err, answer) { var res; if (err) { return gen.throw(err); } else { res = gen.next(answer); } if (!res.done) { res.value(next); } } next(); } run(function* () { try{ var tpContent = yield readFile('blog_post_template.html'); var mdContent = yield readFile('my_blog_post.md'); console.log(tpContent, mdContent); }catch(e){ console.error(e.message); } });
8.34.5. async function
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
- async function returns AsyncFunction object which is Promise
awaitexpressions make promise-returning functions behave as though they're synchronous by suspending execution until the returned promise is resolved.- Deals with one and only one Promise
- Within regular JavaScript code,
awaitis only valid inside async functions.awaitcan be used on its own with JavaScript modules.
- To wait for all promises to finish, use Promise.all
The purpose of
async/awaitis to simplify the syntax necessary to consume promise-based APIs. The behavior ofasync/awaitis similar to combining generators and promises.function resolveAfter2Seconds(x) { return new Promise(resolve => { setTimeout(() => { resolve(x); }, 2000); }); } async function add1(x) { // start and await Promise in order const a = await resolveAfter2Seconds(20); const b = await resolveAfter2Seconds(30); return x + a + b; } add1(10).then(v => { console.log(v); // prints 60 after 4 seconds. }); async function add2(x) { // start Promise all together const p_a = resolveAfter2Seconds(20); const p_b = resolveAfter2Seconds(30); // await each Promise return x + await p_a + await p_b; } add2(10).then(v => { console.log(v); // prints 60 after 2 seconds. });
8.34.6. js:for…of
8.34.7. Import, export
8.34.7.1. ES5
// App.js var React = require('react'); var Component = React.Component; require('./App.css'); var Color = require('./Shapes'); // color component class App extends Component { render() { return ( <div className="App"> <div className="colors"> <Color name="red"/> <Color name="green"/> <Color name="blue"/> </div> </div> ); } } module.exports = App; // Shapes.js var React = require('react'); var Component = React.Component; class Color extends Component { render() { const divStyle = { backgroundColor: this.props.name, color: 'white', fontSize: '20px', height: '100px', width: '100px' } return ( <div style={divStyle}>{this.props.name}</div> ) } } module.exports = Color; // Export multiple components class Animal extends Component { render() { const divStyle = { fontSize: '20px', height: '100px', width: '100px', border: '1px solid black', borderRadius: '50%' } return ( <div style={divStyle}>{this.props.name}</div> ) } } module.exports = { Color: Color, Animal: Animal } // Multiple exports, the import needs to change, too // App.js var {Color} = require('./Shapes'); var {Animal} = require('./Shapes');
8.34.7.2. ES6
//App.js import React, {Component} from 'react'; // React is the default exported from react (react.default) but Component is react.Component import './App.css'; // Color is Shapes.Color not Shapes.default. Same for Animal import {Color, Animal} from './Shapes'; // ... // this way (called default import) is faster import Button from '@mui/material/Button'; // than (called named import) import { Button } from '@mui/material'; // this is slow export default App; //Shapes.js import React, {Component} from 'react'; export class Color extends Component { //... } export class Animal extends Component { //... }
8.34.8. Trailing commas
- Safe in
- array
- object literals since ES5
- function parameters since ES8-2017
- Disallow in
- JSON
8.35. TypeScript
- Playground
- https://www.typescriptlang.org/play/index.html
- (no term)
Basics
npm install -g typescript tsc --version # compile a file to greeter.js tsc greeter.ts # when file path is defined, tsconfig.json is not used by default # change target tsc --target es5 greeter.ts
8.35.1. tsconfig.json
# to create a tsconfig.json tsc --init # to compile based on tsconfig.json tsc -p . # to compile other project tsc -p other/tsconfig.json
8.35.2. Gulp
npm init npm install -g gulp-cli npm i -D typescript gulp@4.0.0 gulp-typescript # add files gulp node dist/main.js
#+NAME ./gulpfile.js
var gulp = require('gulp'); var ts = require('gulp-typescript'); var tsProject = ts.createProject('tsconfig.json'); gulp.task('default', function () { return tsProject.src() .pipe(tsProject()) .js.pipe(gulp.dest('dist')); });
#+NAME src/greet.ts
export function sayHello(name: string) { return `Hello from ${name}`; }
#+NAME src/main.ts
import { sayHello } from './greet'; console.log(sayHello('TypeScript'));
#+NAME ./tsconfig.json
{
"files": [
"src/main.ts",
"src/greet.ts"
],
"compilerOptions": {
"noImplicitAny": true,
"target": "es5"
}
}
- With Browserify
npm i -D browserify tsify vinyl-source-stream- tsify is a Browserify plugin
- gives access to TypeScript compiler
- viny-source-stream
- adapt the file output of Browserify to Gulp format called vinyl
- (no term)
Example #+NAME ./src/index.html
<!DOCTYPE html> <html> <head> <meta charset="UTF-8" /> <title>Hello World!</title> </head> <body> <p id="greeting">Loading ...</p> <script src="bundle.js"></script> </body> </html>
#+NAME ./src/main.ts
import { sayHello } from './greet'; function showHello(divName: string, name: string) { const elt = document.getElementById(divName); elt.innerText = sayHello(name); } showHello('greeting', 'TypeScript');
#+NAME gulpfile.js
var gulp = require('gulp'); var browserify = require('browserify'); var source = require('vinyl-source-stream'); var tsify = require('tsify'); var paths = { pages: ['src/*.html'] }; gulp.task('copy-html', function () { return gulp.src(paths.pages) .pipe(gulp.dest('dist')); }); gulp.task('default', gulp.series(gulp.parallel('copy-html'), function () { return browserify({ basedir: '.', debug: true, entries: ['src/main.ts'], cache: {}, packageCache: {} }) .plugin(tsify) .bundle() .pipe(source('bundle.js')) .pipe(gulp.dest('dist')); }));
- (no term)
With Watchify
npm i -D watchify fancy-log#+NAME ./gulpfile.jsvar gulp = require('gulp'); var browserify = require('browserify'); var source = require('vinyl-source-stream'); var watchify = require('watchify'); var tsify = require('tsify'); var fancy_log = require('fancy-log'); var paths = { pages: ['src/*.html'] }; var watchedBrowserify = watchify(browserify({ basedir: '.', debug: true, entries: ['src/main.ts'], cache: {}, packageCache: {} }).plugin(tsify)); gulp.task('copy-html', function () { return gulp.src(paths.pages) .pipe(gulp.dest('dist')); }); function bundle() { return watchedBrowserify .bundle() .on('error', fancy_log) .pipe(source('bundle.js')) .pipe(gulp.dest('dist')); } gulp.task('default', gulp.series(gulp.parallel('copy-html'), bundle)); watchedBrowserify.on('update', bundle); watchedBrowserify.on('log', fancy_log);
- (no term)
With Uglify
npm i -D gulp-uglify vinyl-buffer gulp-sourcemaps#+NAME ./gulpfile.jsvar gulp = require('gulp'); var browserify = require('browserify'); var source = require('vinyl-source-stream'); var tsify = require('tsify'); var uglify = require('gulp-uglify'); var sourcemaps = require('gulp-sourcemaps'); var buffer = require('vinyl-buffer'); var paths = { pages: ['src/*.html'] }; gulp.task('copy-html', function () { return gulp.src(paths.pages) .pipe(gulp.dest('dist')); }); gulp.task('default', gulp.series(gulp.parallel('copy-html'), function () { return browserify({ basedir: '.', debug: true, entries: ['src/main.ts'], cache: {}, packageCache: {} }) .plugin(tsify) .bundle() .pipe(source('bundle.js')) .pipe(buffer()) // new .pipe(sourcemaps.init({loadMaps: true})) // new .pipe(uglify()) // new .pipe(sourcemaps.write('./')) // new .pipe(gulp.dest('dist')); }));
- (no term)
With Babel
npm i -D babelify@8 babel-core babel-preset-es2015 vinyl-buffer gulp-sourcemapsChange TypeScript to output ES2015 as target. Later Babel will produce ES5 from it. #+NAME tsconfig.json{ "files": [ "src/main.ts" ], "compilerOptions": { "noImplicitAny": true, "target": "es2015" } }#+NAME ./gulpfile.js
var gulp = require('gulp'); var browserify = require('browserify'); var source = require('vinyl-source-stream'); var tsify = require('tsify'); var sourcemaps = require('gulp-sourcemaps'); var buffer = require('vinyl-buffer'); var paths = { pages: ['src/*.html'] }; gulp.task('copy-html', function () { return gulp.src(paths.pages) .pipe(gulp.dest('dist')); }); gulp.task('default', gulp.series(gulp.parallel('copy-html'), function () { return browserify({ basedir: '.', debug: true, entries: ['src/main.ts'], cache: {}, packageCache: {} }) .plugin(tsify) // output is files of es2015 standard .transform('babelify', { presets: ['es2015'], extensions: ['.ts'] }) // new. Modify Babelify setting: add .ts extension for file processing .bundle() .pipe(source('bundle.js')) .pipe(buffer()) // new .pipe(sourcemaps.init({loadMaps: true})) // new .pipe(sourcemaps.write('./')) // new .pipe(gulp.dest('dist')); }));
8.36. Use Cases
8.36.1. Click to anchor
Changes the hash in URL and then go to the element
<div onclick="goToAnchor('gotoanchor-section-name')">
something here
</div>
<div class="spacer" id="gotoanchor-section-name"></div>
<div> Content I want to go to </div>
<script>
function goToAnchor(anchor) {
var loc = document.location.toString().split('#')[0];
document.location = loc + '#' + anchor;
return false;
}
</script>
8.36.2. Get radio value, Event on radio buttons
<input type="radio" name="genderS" value="1" checked onclick="console.log('value is 1');"> Male <input type="radio" name="genderS" value="0" onclick="console.log('value is 0');"> Female
// Returns the checked radio value, if no radio is checked var checkedRadio = document.querySelector('input[name="my_radio_name"]:checked'); if (checkedRadio) { console.log(checkedRadio.value); } else { console.log('No radio is checked'); } // Old function getRadioValue(name) { var radios = document.getElementsByName(name); if (!!radios && radios.length > 0) { for (var i = 0, length = radios.length; i < length; i++) { if (radios[i].checked) { return radios[i].value; // only one radio can be logically checked, don't check the rest } } } return null; } getRadioValue('genderS');
8.36.3. Detect iOS version
https://gist.github.com/Craga89/2829457
/* * Outputs a float representing the iOS version if user is using an iOS browser i.e. iPhone, iPad * Possible values include: * 3 - v3.0 * 4.0 - v4.0 * 4.14 - v4.1.4 * false - Not iOS */ var iOS = parseFloat( ('' + (/CPU.*OS ([0-9_]{1,5})|(CPU like).*AppleWebKit.*Mobile/i.exec(navigator.userAgent) || [0,''])[1]) .replace('undefined', '3_2').replace('_', '.').replace('_', '') ) || false;
9. JavaScript projects
9.1. Vue.js
9.1.1. Basics
- Who use it?
- Builds
- https://cdn.jsdelivr.net/npm/vue is the Runtime + Compiler UMD Minimized latest build
- Runtime only is good for webpack.
- https://unpkg.com/vue is the same above but not minimized (development)
- All latest builds
- https://cdn.jsdelivr.net/npm/vue is the Runtime + Compiler UMD Minimized latest build
Example An instance is created also called viewmodel in
new Vue()<script src="https://unpkg.com/vue"></script> <div id="app"> <!-- bind attribute to a var --> <span v-bind:title="msg2"> {{ message }} </span> <div> <span v-if="seen">Now you see me</span> </div> <div> <ol> <li v-for="todo in todos"> {{ todo.text }} </li> </ol> </div> <button v-on:click="reverseMessage">Reverse Message</button> <!-- 2 way binding --> <div> <input v-model="message"> </div> </div> <script> var my_vue_app = new Vue({ el: '#app', data: { message: 'Hello Vue.js!', msg2: 'span title', seen: true, // my_vue_app.todos.push({ text: 'New item' }) todos: [ { text: 'Learn JavaScript' }, { text: 'Learn Vue' }, { text: 'Build something awesome' } ] }, methods: { reverseMessage: function () { this.message = this.message.split('').reverse().join('') } } }) </script>
9.1.2. Data and methods vm.data vue:data
Root-level reactive properties need to be defined when the instance was first created to ensure reactive behavior (trigger view updates)
var data = {a:1} var vm = new Vue({ data: data }); vm.a === data.a // true, they refer to the same object vm.a = 2 // then data.a is 2, too data.a = 3 // and vice-versa vm.a === 3 vm.b = 'hi' // the properties in data are active only when the instance was first created // a new property is defined after initialization, so it won't trigger view update // Define all variables with default values you need in initialization
Special instance properties and methods
var data = { a: 1 } var vm = new Vue({ el: '#example', data: data }) vm.$data === data // => true vm.$el === document.getElementById('example') // => true // $watch is an instance method vm.$watch('a', function (newValue, oldValue) { // This callback will be called when `vm.a` changes })
- Some plugins allow components to access/change specific root-level properties e.g.
this.$storevue:plugin:vuex,this.$routervue:router - Don't use arrow function
9.1.2.1. Mutation, Vue.set vue:data:mutation
- Mutation methods
- when these methods are called, the views will be updated
- push, pop, shift, unshift, splice, sort, reverse
- e.g.
my_vue_app.items.push({ message: 'Baz'});
- (no term)
Non-mutating methods always return a new array e.g. filter, concat
example1.items = example1.items.filter(function (item) { return item.message.match(/Foo/) })
- Add new prop reactivitely
Don't use
vm.items[indexOfItem] = newValue, instead use:// Vue.set Vue.set(example1.items, indexOfItem, newValue) // Or // Array.prototype.splice example1.items.splice(indexOfItem, 1, newValue)
Don't use vm.items.length = newLenth, instead use:
example1.items.splice(newLength)
Add reactive properties to an object at root-level
var vm = new Vue({ data: { userProfile: { name: 'Anika' } } }) Vue.set(vm.userProfile, 'age', 27); // or this.$set(this.userProfile, 'age', 27) // or vm.$set
The other way to add a new prop is to rebuild the object using Object.assign() or _.extend()
/* Don't do it directly Object.assign(this.userProfile, { age: 27, favoriteColor: 'Vue Green' }) */ this.userProfile = Object.assign({}, this.userProfile, { age: 27, favoriteColor: 'Vue Green' })
9.1.3. Instance Lifecycle Hooks
- Data observation > compile templates > mount instance to DOM > Update DOM when data changes
- Don't use arrow function in those hooks e.g.
created: () => console.log(this.a)- this is ok
mounted() => {}
- this is ok
- Full lifecycle diagram and hooks
9.1.3.1. beforeCreate, created
run code after an instance is created
new Vue({ data: { a: 1 }, created: function () { // `this` points to the vm instance console.log('a is: ' + this.a) } }) // => "a is: 1"
9.1.3.2. vm.$mount(el), beforeMount, mounted
9.1.3.3. beforeUpdate, updated
9.1.3.4. beforeDestroy, destroyed
9.1.4. Component
9.1.4.1. Basics
Example
<div id="app-7"> <ol> <!-- Now we provide each todo-item with the todo object it's representing, so that its content can be dynamic. We also need to provide each component with a "key", which will be explained later. --> <todo-item v-for="item in groceryList" v-bind:todo="item" v-bind:key="item.id"> </todo-item> </ol> </div>
Vue.component('todo-item', { props: ['todo'], template: '<li>{{ todo.text }}</li>' }) var app7 = new Vue({ el: '#app-7', data: { groceryList: [ { id: 0, text: 'Vegetables' }, { id: 1, text: 'Cheese' }, { id: 2, text: 'Whatever else humans are supposed to eat' } ] } })
Vue.componentregisters a component globally. Make a component available only in the scope of another instance/component by registering it withcomponentsinstance optionvar Child = { template: '<div>A custom component!</div>' } new Vue({ // ... components: { // <my-component> will only be available in parent's template 'my-component': Child } })
- all lowercase and must contain a hyphen
Although these can be used in string templates
components: { 'kebab-cased-component': { /* ... */ }, camelCasedComponent: { /* ... */ }, PascalCasedComponent: { /* ... */ } }
Component should be used in a way that has 3 parts
- props
- pass from parent to component
- events
- set parent based on events triggered from component
- slots
- customize, from parent, the look of the component
<my-component :foo="baz" :bar="qux" @event-a="doThis" @event-b="doThat" > <img slot="icon" src="..."> <p slot="main-text">Hello!</p> </my-component>
9.1.4.2. DOM Template Parsing Caveats
Becuase <ul>, <ol>, <table> and <select> need to have certain elements appearing inside them and some elements such as <option> can only appear inside certain other elements, it will lead to parsing issues if you do this
<table> <my-row>...</my-row> </table>
So better to do this
<table> <tr is="my-row"></tr> </table>
camelCase vs. kebab-case
These limitations do not apply if you are using string templates from one of the following sources:
- <script type='text/x-template'>
- JavaScript inline template strings
- .vue components
So, prefer using string templates whenever possible.
9.1.4.3. Option data has to be a function
data is the initialization part that defines extra local data properties that are only available in the component.
3 simple-counter's have their own initializing part and scopes.
<div id="example-2">
<simple-counter></simple-counter>
<simple-counter></simple-counter>
<simple-counter></simple-counter>
</div>
Vue.component('simple-counter', {
template: '<button v-on:click="counter += 1">{{ counter }}</button>',
data: function () {
return {
counter: 0
}
}
})
new Vue({
el: '#example-2'
})
9.1.4.4. Option filters
- Filters provide text formatting
<!-- in mustaches --> {{ message | capitalize }} <!-- in v-bind --> <div v-bind:id="rawId | formatId"></div> <!-- can be chained --> {{ message | filterA | filterB }} <!-- can have arguments --> {{ message | filterA('arg1', arg2) }}
filters: {
capitalize: function (value) {
if (!value) return ''
value = value.toString()
return value.charAt(0).toUpperCase() + value.slice(1)
}
}
// define globally
Vue.filter('capitalize', function (value) {
if (!value) return ''
value = value.toString()
return value.charAt(0).toUpperCase() + value.slice(1)
})
9.1.4.5. Props down
Pass down prop from parent to child using child component's prop option.
When the parent property updates, it will flow down to the child but not the other way around.
Vue.component('child', {
// declare the props
props: ['message'],
// like data, the prop can be used inside templates and
// is also made available in the vm as this.message
template: '<span>{{ message }}</span>'
})
// parent page
<child message="hello!"></child>
// dynamic binding a parent variable
<div>
<input v-model="parentMsg">
<br>
<child v-bind:my-message="parentMsg"></child>
</div>
// pass all properties of the parent as props to the child
todo: {
text: 'Learn Vue',
isComplete: false
}
<todo-item v-bind="todo"></todo-item>
<todo-item
v-bind:text="todo.text"
v-bind:is-complete="todo.isComplete"
></todo-item>
Literal vs Dynamic
<!-- this passes down a plain string "1" --> <comp some-prop="1"></comp> <!-- this passes down an actual number --> <comp v-bind:some-prop="1"></comp>
- Don't mutate a prop of child component
You shouldn't mutate a prop inside a child component. Refer to vue:data:mutation
Because objects and arrays in JavaScript are passed by reference. So mutating the object or array inside the child will affect parent state.
Instead you should do the following:
- Define a local data prop that uses the prop's initial value as its initial value
props: ['initialCounter'], data: function () { return { counter: this.initialCounter } }- Define a computed prop that is computed from the prop's value
props: ['size'], computed: { normalizedSize: function () { return this.size.trim().toLowerCase() } } - prop validation
When prop validation fails, Vue will produce a console warning (if using the development build). Note that props are validated before a component instance is created, so within default or validator functions, instance properties such as from data, computed, or methods will not be available.
Vue.component('example', { props: { // basic type check (`null` means accept any type) propA: Number, // multiple possible types propB: [String, Number], // a required string propC: { type: String, required: true }, // a number with default value propD: { type: Number, default: 100 }, // object/array defaults should be returned from a // factory function propE: { type: Object, default: function () { return { message: 'hello' } } }, // custom validator function propF: { validator: function (value) { return value > 10 } } } })
9.1.4.6. Events Up
9.1.4.7. DOM template
- In non-string templates, camelCased prop names need to use their kebab-case equivalents
Refer to see what vue:string template is
Vue.component('child', { // camelCase in JavaScript props: ['myMessage'], template: '<span>{{ myMessage }}</span>' })
<!-- kebab-case in HTML --> <child my-message="hello!"></child>
9.1.4.8. Except class and style :: parent attributes overwrite child component
Child component bs-date-input has this template <input type="date" class="form-control">
But the child component is called from parent with the type attribute as well.
<bs-date-input data-3d-date-picker="true" class="date-picker-theme-dark" type="large" ></bs-date-input>
The final result is type="large" because the value provided to the component will replace the value set by the component.
Except class and style attributes.
9.1.4.9. Slots
Parent content will be discarded unless the child component template contains at least one <slot> outlet.
- Basics
my-component template:
<div> <h2>I'm the child title</h2> <slot> This will only be displayed if there is no content to be distributed. </slot> </div>the parent
<div> <h1>I'm the parent title</h1> <my-component> <p>This is some original content</p> <p>This is some more original content</p> </my-component> </div>Result
<div> <h1>I'm the parent title</h1> <div> <h2>I'm the child title</h2> <p>This is some original content</p> <p>This is some more original content</p> </div> </div> - Multiple slots
app-layout component with template
<div class="container"> <header> <slot name="header"></slot> </header> <main> <slot></slot> </main> <footer> <slot name="footer"></slot> </footer> </div>
Parent
<app-layout> <h1 slot="header">Here might be a page title</h1> <p>A paragraph for the main content.</p> <p>And another one.</p> <p slot="footer">Here's some contact info</p> </app-layout>
Result
<div class="container"> <header> <h1>Here might be a page title</h1> </header> <main> <p>A paragraph for the main content.</p> <p>And another one.</p> </main> <footer> <p>Here's some contact info</p> </footer> </div>
- Scoped slot
From parent, create a temporary variable with an alias name which holds the props object passed from the child (props.text defined in child)
This provides a way for parent to customize the look of a component.
child component
<div class="child"> <slot text="hello from child"></slot> </div>
Parent
<div class="parent"> <child> <template slot-scope="props"> <span>hello from parent</span> <span>{{ props.text }}</span> </template> </child> </div>Result
<div class="parent"> <div class="child"> <span>hello from parent</span> <span>hello from child</span> </div> </div>In 2.5.0+, slot-scope is no longer limited to <template> and can be used on any element or component.
Number of slots in parent do not need to match number of slots in component.
Parent :: customize the look for each list item
<my-awesome-list :items="items"> <!-- scoped slot can be named too --> <li slot="item" slot-scope="props" class="my-fancy-item"> {{ props.text }} </li> </my-awesome-list>my-awesome-list component template
<ul> <slot name="item" v-for="item in items" :text="item.text"> <!-- fallback content here --> </slot> </ul> - Destructuring for slot-scope
<child> <span slot-scope="{ text }">{{ text }}</span> </child>
9.1.4.10. Dynamic component, v-bind:is, <component>, <keep-alive>
var vm = new Vue({ el: '#example', data: { currentView: 'home' }, components: { home: { /* ... */ }, posts: { /* ... */ }, archive: { /* ... */ } } })
<component v-bind:is="currentView"> <!-- component changes when vm.currentView changes! --> </component>
If <transition> is used, wrap <keep-alive> inside <transition>
<keep-alive> <component :is="currentView"> <!-- inactive components will be cached! --> </component> </keep-alive>
9.1.4.11. Get Child Component One Time
$refs are only populated after the component has been rendered and it is not reactive.
<div id="parent">
<user-profile ref="profile"></user-profile>
</div>
var parent = new Vue({ el: '#parent' })
// access child component instance
var child = parent.$refs.profile
9.1.4.12. Async Components
Define component as a factory function that asynchronously resolves the component definition. The factory function will only be triggered when the component actually needs to be rendered and will cache the result for future re-renders.
Usually, component is defined using a name and an object of props.
A factory function can be used:
Vue.component('async-example', function (resolve, reject) {
setTimeout(function () {
// Pass the component definition to the resolve callback
resolve({
template: '<div>I am async!</div>'
})
}, 1000)
})
Webpack example
Vue.component('async-webpack-example', function (resolve) {
// This special require syntax will instruct Webpack to
// automatically split your built code into bundles which
// are loaded over Ajax requests.
require(['./my-async-component'], resolve)
})
- Component factory function can return a Promise
(with Webpack)
Vue.component( 'async-webpack-example', // The `import` function returns a `Promise`. () => import('./my-async-component') ) new Vue({ // ... components: { 'my-component': () => import('./my-async-component') } }) - Component factory function can also return an object
const AsyncComp = () => ({ // The component to load. Should be a Promise component: import('./MyComp.vue'), // A component to use while the async component is loading loading: LoadingComp, // A component to use if the load fails error: ErrorComp, // Delay before showing the loading component. Default: 200ms. delay: 200, // The error component will be displayed if a timeout is // provided and exceeded. Default: Infinity. timeout: 3000 })Refer to vue:router
9.1.4.13. Recursive and Circular Reference
A component can recursively invoke itself in its own template
When a component is registered globally using Vue.component, the global ID is auto set as the component's name option
Vue.component('unique-name-of-my-component', {
// ...
})
name: 'unique-name-of-my-component'
- Recursive
name: 'stack-overflow', template: '<div><stack-overflow></stack-overflow></div>
- Circular reference
tree-folder component
<p> <span>{{ folder.name }}</span> <tree-folder-contents :children="folder.children"/> </p>tree-folder-contents component
<ul> <li v-for="child in children"> <tree-folder v-if="child.children" :folder="child"/> <span v-else>{{ child.name }}</span> </li> </ul>tree-folder needs tree-folder-contents and tree-folder-contents needs tree-folder.
It's ok if no module system is used to import components (Webpack or Browserify).
You should do this
beforeCreate: function () { this.$options.components.TreeFolderContents = require('./tree-folder-contents.vue') }
9.1.4.14. inline-template vue:inline-template
It makes the inner content as the component's template. Don't use it
<my-component inline-template> <div> <p>These are compiled as the component's own template.</p> <p>Not parent's transclusion content.</p> </div> </my-component>
9.1.4.15. X-Templates vue:x-template
Don't use
<script type="text/x-template" id="hello-world-template"> <p>Hello hello hello</p> </script>
Vue.component('hello-world', { template: '#hello-world-template' })
9.1.5. Template Syntax
9.1.5.1. String template vs DOM template
String template vue:string template
Vue.component('my-component', { template: '<p>This is a String template, it iss passed as a string to the component.</p>' })
DOM template
<body> <div id="app"> <!-- your App is runnning in this div ---> <my-component></my-component> </div> <template id="template-for-my-component"> {{ message }} </template> </body>
- vue:x-template
9.1.5.2. v-once
one-time interpolations and remain cached
<span v-once>This will never change: {{ msg }}</span>
Vue.component('terms-of-service', { template: '\ <div v-once>\ <h1>Terms of Service</h1>\ ... a lot of static content ...\ </div>\ ' })
9.1.5.3. v-html Raw HTML
Can't be used for components. Watch out for XSS
<div v-html="rawHtml"></div>
9.1.5.4. v-bind:attributeName, :attributeName
<!-- bool is a bit different. If isButtonDisabled has the value of null, undefined or false, the disabled attribute will not be rendered --> <button v-bind:disabled="isButtonDisabled">Button</button> <button :disabled="isButtonDisabled">Button</button> <!-- multiple attributes binding --> <img v-bind="{ class: imgClass, src: products[2].image, alt: products[2].alt }">
9.1.5.5. Expression
- Some JavaScript expressions are allowed
- search
allowedGlobalsin Vue Core - (no term)
- Evaluated in the data scope of the owner Vue instance
- (no term)
- One binding one expression!
- (no term)
- Don't access user defined globals in template expressions
- (no term)
- Value of a directive attribute
v-*usually is a single JavaScript expression
{{ number + 1 }}
{{ ok ? 'YES' : 'NO' }}
{{ message.split('').reverse().join('') }}
<div v-bind:id="'list-' + id"></div>
9.1.5.6. Directive Arguments v-bind:href, v-bind:class, v-on:click
- v-bind:href, v-on:click
<a v-bind:href="url"> ... </a> // bind element's href attribute to the value of the expression url <a v-on:click="doSomething"> ... </a> // shorthand <a @click="doSomething">...</a>
v-bind:class
<div class="static" v-bind:class="{ active: isActive, 'text-danger': hasError }"> </div> <!-- <div class="static active"></div> -->
data: { isActive: true, hasError: false }Or
<div v-bind:class="classObject"></div>
data: { classObject: { active: true, 'text-danger': false } } // or even data: { isActive: true, error: null }, computed: { classObject: function () { return { active: this.isActive && !this.error, 'text-danger': this.error && this.error.type === 'fatal' } } }Array
<div :class="[activeClass, errorClass]"></div> <!-- data: { activeClass: 'active', errorClass: 'text-danger' } --> <div v-bind:class="[isActive ? activeClass : '', errorClass]"></div> <div v-bind:class="[{ active: isActive }, errorClass]"></div>
When you use the class attribute on a custom component, those classes will be added to the component’s root element. Existing classes on this element will not be overwritten.
Vue.component('my-component', { template: '<p class="foo bar">Hi</p>' }) // <my-component class="baz boo"></my-component> // the rendered HTML will be: // <p class="foo bar baz boo">Hi</p> // the same is also true for class bindings: // <my-component v-bind:class="{ active: isActive }"></my-component> // Rendered: // <p class="foo bar active">Hi</p>
v-bind:style
- CSS vendor-prefixes are auto detected and added when
v-bind:styleis used - Use either camelCase with hypen and quotes or kebab-case with hyphen and quotes for CSS property name
<div v-bind:style="{ color: activeColor, fontSize: fontSize + 'px' }"></div> <!-- kebob case --> <div v-bind:style="{ color: activeColor, 'font-size': fontSize + 'px' }"></div> <!-- data: { activeColor: 'red', fontSize: 30 } --> <div v-bind:style="styleObject"></div> <!-- data: { styleObject: { color: 'red', fontSize: '13px' } } -->
Array
<div v-bind:style="[baseStyles, overridingStyles]"></div>
- CSS vendor-prefixes are auto detected and added when
9.1.5.7. Dynamic Directive Argument
myAttributeNameVar- expression should return string or null. Any other non-string value triggers a warning
- null
- remove the binding
- Don't use spaces or quotes in the expression
- DON'T
<a v-bind:['foo' + bar]="value"> ... </a> - In-DOM templates (templates in an HTML file)
- expression will be converted to lowercase
<a v-bind:[myAttributeNameVar]="url"> ... </a> <a @[myeventnamevar]="doSomething">...</a>
9.1.5.8. Modifiers v-on:submit.prevent
- Refer to vue:event:modifiers
Example Calls
event.preventDefault()on the triggered event<form v-on:submit.prevent="onSubmit"> ... </form>
9.1.5.9. v-if, key attribute, v-show
v-if,v-else,v-else-if<h1 v-if="ok">Yes</h1> <h1 v-else>No</h1>
v-elsemust immediately follow av-iforv-else-ifelement<div v-if="Math.random() > 0.5"> Now you see me </div> <div v-else> Now you don't </div>
v-else-ifsince 2.1.0+<div v-if="type === 'A'"> A </div> <div v-else-if="type === 'B'"> B </div> <div v-else-if="type === 'C'"> C </div> <div v-else> Not A/B/C </div>
- When used together with
v-if,v-forhas a higher priority thanv-if. Group with
v-ifon<template>. template element will not be included<template v-if="ok"> <h1>Title</h1> <p>Paragraph 1</p> <p>Paragraph 2</p> </template>
keyattribute marks an element unique which means elements shouldn't be reused unless it has the same key<template v-if="loginType === 'username'"> <label>Username</label> <input placeholder="Enter your username" key="username-input"> </template> <template v-else> <label>Email</label> <input placeholder="Enter your email address" key="email-input"> </template>
v-showtoggles the element’sdisplayCSS property based on the truthy-ness of the expression valuev-showdoesn’t support the<template>element, nor does it work withv-else.v-ifvsv-show- Expensive to toggle
v-if, so use it when the condition is unlikely to change over time v-ifis lazy means if it's false on initial render, it will not do anything, the conditional block won't be rendered until the condition becomes true for the first timev-showalways render- Use
v-showif it needs to be toggled very often
- Expensive to toggle
<h1 v-show="ok">Hello!</h1>
9.1.5.10. v-for
- Basics
v-for has full access to parent scope properties
// array <ul id="example-1"> <li v-for="item in items"> {{ item.message }} </li> </ul> var example1 = new Vue({ el: '#example-1', data: { items: [ { message: 'Foo' }, { message: 'Bar' } ] } }) <ul id="example-2"> <li v-for="(item, index) in items"> {{ parentMessage }} - {{ index }} - {{ item.message }} </li> </ul> var example2 = new Vue({ el: '#example-2', data: { parentMessage: 'Parent', items: [ { message: 'Foo' }, { message: 'Bar' } ] } }) <div v-for="item of items"></div> // object <ul id="v-for-object" class="demo"> <li v-for="value in object"> {{ value }} </li> </ul> new Vue({ el: '#v-for-object', data: { object: { firstName: 'John', lastName: 'Doe', age: 30 } } }) <div v-for="(value, key) in object"> {{ key }}: {{ value }} </div> <div v-for="(value, key, index) in object"> {{ index }}. {{ key }}: {{ value }} </div>The default mode is only suitable when your list render output does not reply on child component state or temporary DOM state (e.g. form input values)
Provide a unqiue key attribute for each item.
<div v-for="item in items" :key="item.id"> <!-- content --> </div>
Refer to vue:data for mutation and non-mutating methods
- Filtered/Sorted
<li v-for="n in evenNumbers">{{ n }}</li> data: { numbers: [ 1, 2, 3, 4, 5 ] }, computed: { evenNumbers: function () { return this.numbers.filter(function (number) { return number % 2 === 0 }) } }In case when computed properties are not feasible e.g. inside nested v-for loops
<li v-for="n in even(numbers)">{{ n }}</li> data: { numbers: [ 1, 2, 3, 4, 5 ] }, methods: { even: function (numbers) { return numbers.filter(function (number) { return number % 2 === 0 }) } } - range
<div> <span v-for="n in 10">{{ n }} </span> </div> - v-for on
<template>
<ul> <template v-for="item in items"> <li>{{ item.msg }}</li> <li class="divider"></li> </template> </ul>
- v-for with v-if
<li v-for="todo in todos" v-if="!todo.isComplete"> {{ todo }} </li> // don't execute v-for at all <ul v-if="todos.length"> <li v-for="todo in todos"> {{ todo }} </li> </ul> <p v-else>No todos left!</p> - v-for with a component
<my-component v-for="item in items" :key="item.id"></my-component> // pass data to component <my-component v-for="(item, index) in items" v-bind:item="item" v-bind:index="index" v-bind:key="item.id" ></my-component>
- Example
Because it has to be at least one <li> in <ul> to be valid.
is="todo-item"is needed.<div id="todo-list-example"> <input v-model="newTodoText" v-on:keyup.enter="addNewTodo" placeholder="Add a todo" > <ul> <li is="todo-item" v-for="(todo, index) in todos" v-bind:key="todo.id" v-bind:title="todo.title" v-on:remove="todos.splice(index, 1)" ></li> </ul> </div> Vue.component('todo-item', { template: '\ <li>\ {{ title }}\ <button v-on:click="$emit(\'remove\')">X</button>\ </li>\ ', props: ['title'] }) new Vue({ el: '#todo-list-example', data: { newTodoText: '', todos: [ { id: 1, title: 'Do the dishes', }, { id: 2, title: 'Take out the trash', }, { id: 3, title: 'Mow the lawn' } ], nextTodoId: 4 }, methods: { addNewTodo: function () { this.todos.push({ id: this.nextTodoId++, title: this.newTodoText }) this.newTodoText = '' } } })
9.1.5.11. Event, v-on, @attributeName
- Basics
<div id="example-1"> <button v-on:click="counter += 1">Add 1</button> <p>The button above has been clicked {{ counter }} times.</p> </div> var example1 = new Vue({ el: '#example-1', data: { counter: 0 } }) - Method calling
<div id="example-2"> <!-- `greet` is the name of a method defined below --> <button v-on:click="greet">Greet</button> </div> var example2 = new Vue({ el: '#example-2', data: { name: 'Vue.js' }, // define methods under the `methods` object methods: { greet: function (event) { // `this` inside methods points to the Vue instance alert('Hello ' + this.name + '!') // `event` is the native DOM event if (event) { alert(event.target.tagName) } } } }) // you can invoke methods in JavaScript too example2.greet() // => 'Hello Vue.js!' - Inline with $event
<div id="example-3"> <button v-on:click="say('hi')">Say hi</button> <button v-on:click="say('what')">Say what</button> </div> new Vue({ el: '#example-3', methods: { say: function (message) { alert(message) } } }) <button v-on:click="warn('Form cannot be submitted yet.', $event)"> Submit </button> methods: { warn: function (message, event) { // now we have access to the native event if (event) event.preventDefault() alert(message) } } - Event Modifiers vue:event:modifiers
<!-- the click event's propagation will be stopped --> <a v-on:click.stop="doThis"></a> <!-- the submit event will no longer reload the page --> <form v-on:submit.prevent="onSubmit"></form> <!-- modifiers can be chained :: chain stop and prevent. --> <a v-on:click.stop.prevent="doThat"></a> <!-- chain order matters. @click.prevent.self will prevent all clicks while @click.self.prevent will only prevent clicks on the element itself --> <!-- just the modifier --> <form v-on:submit.prevent></form> <!-- use capture mode when adding the event listener --> <!-- i.e. an event targeting an inner element is handled here before being handled by that element --> <div v-on:click.capture="doThis">...</div> <!-- only trigger handler if event.target is the element itself --> <!-- i.e. not from a child element --> <div v-on:click.self="doThat">...</div> <!-- the click event will be triggered at most once. 2.1.4+ --> <a v-on:click.once="doThis"></a>
- Event Key Modifiers vue:event:key modifiers
<!-- only call vm.submit() when the keyCode is 13 --> <input v-on:keyup.13="submit"> <!-- also works for shorthand --> <input @keyup.enter="submit">
Full list :: .enter, .tab, .delete (both Delete and Backspace), .esc, .space, .up, .down, .left, .right
Define custom key modifier aliase
// enable v-on:keyup.f1 Vue.config.keyCodes.f1 = 112
Directly use any valid key names provided by KeyboardEvent.key as modifiers by converting them to kebab-case. e.g. the handler will only be called if $event.key
='PageDown'.<input @keyup.page-down="onPageDown">
.ctrl, .alt, .shift, .meta (windows key or Mac's command key)
<!-- Alt + C --> <input @keyup.alt.67="clear"> <!-- Ctrl + Click --> <div @click.ctrl="doSomething">Do something</div>
.exact
<!-- this will fire even if Alt or Shift is also pressed --> <button @click.ctrl="onClick">A</button> <!-- this will only fire when only Ctrl is pressed --> <button @click.ctrl.exact="onCtrlClick">A</button>
.left, .right, .middle
- Event from child component. .sync vue:event:component
<div id="counter-event-example"> <p>{{ total }}</p> <button-counter v-on:increment="incrementTotal"></button-counter> <button-counter v-on:increment="incrementTotal"></button-counter> </div> Vue.component('button-counter', { template: '<button v-on:click="incrementCounter">{{ counter }}</button>', data: function () { return { counter: 0 } }, methods: { incrementCounter: function () { this.counter += 1 this.$emit('increment') } }, }) new Vue({ el: '#counter-event-example', data: { total: 0 }, methods: { incrementTotal: function () { this.total += 1 } } })vm.$on can only listen events on teh current vm. But in order to listen from child components, you will need v-on in parent template
vm.$on('test', function (msg) { console.log(msg) }) vm.$emit('test', 'hi') // => "hi"In parent, when adding a native even (e.g. click) on a component, @click.native="parentMethod" is required.
.sync By default one-way data flow from parent to child. There's time when modifying parent prop is desired. But try to avoid this.
<comp :foo.sync="bar"></comp> // got expanded to <comp :foo="bar" @update:foo="val => bar = val"></comp> // For the child component to update foo‘s value, it needs to explicitly emit an event instead of mutating the prop: this.$emit('update:foo', newValue) - Event in Form, v-model vue:event:v-model
<input v-model="something"> // is equivalent of <input v-bind:value="something" v-on:input="something = $event.target.value"> // For a component <custom-input :value="something" @input="value => { something = value }"> </custom-input>Custom form input The key point is the component has to have props: ['value'] and emit an input event with the new value
- text
<script src="https://cdn.rawgit.com/chrisvfritz/5f0a639590d6e648933416f90ba7ae4e/raw/974aa47f8f9c5361c5233bd56be37db8ed765a09/currency-validator.js"></script> <div id="app"> <currency-input label="Price" v-model="price" ></currency-input> <currency-input label="Shipping" v-model="shipping" ></currency-input> <currency-input label="Handling" v-model="handling" ></currency-input> <currency-input label="Discount" v-model="discount" ></currency-input> <p>Total: ${{ total }}</p> </div> Vue.component('currency-input', { template: '\ <div>\ <label v-if="label">{{ label }}</label>\ $\ <input\ ref="input"\ v-bind:value="value"\ v-on:input="updateValue($event.target.value)"\ v-on:focus="selectAll"\ v-on:blur="formatValue"\ >\ </div>\ ', props: { value: { type: Number, default: 0 }, label: { type: String, default: '' } }, mounted: function () { this.formatValue() }, methods: { updateValue: function (value) { var result = currencyValidator.parse(value, this.value) if (result.warning) { this.$refs.input.value = result.value } this.$emit('input', result.value) }, formatValue: function () { this.$refs.input.value = currencyValidator.format(this.value) }, selectAll: function (event) { // Workaround for Safari bug // http://stackoverflow.com/questions/1269722/selecting-text-on-focus-using-jquery-not-working-in-safari-and-chrome setTimeout(function () { event.target.select() }, 0) } } }) new Vue({ el: '#app', data: { price: 0, shipping: 0, handling: 0, discount: 0 }, computed: { total: function () { return (( this.price * 100 + this.shipping * 100 + this.handling * 100 - this.discount * 100 ) / 100).toFixed(2) } } }) - checkbox
By default, v-model on a component uses value as the prop and input as the event, but some input types such as checkboxes and radio buttons may want to use the value prop for a different purpose. Using the model option can avoid the conflict in such cases
Vue.component('my-checkbox', { model: { prop: 'checked', event: 'change' }, props: { checked: Boolean, // this allows using the `value` prop for a different purpose value: String }, // ... }) <my-checkbox v-model="foo" value="some value"></my-checkbox> // is equivalent to <my-checkbox :checked="foo" @change="val => { foo = val }" value="some value"> </my-checkbox>
- text
- Use Event to transfer data between components that are not parent-child
var bus = new Vue() bus.$emit('id-selected', 1) // in component B's created hook bus.$on('id-selected', function (id) { // ... })If it's more complex, refer to vue:state management
9.1.5.12. Form input, v-model
v-model will ignore the initial value, checked or selected.
- textarea
<span>Multiline message is:</span> <p style="white-space: pre-line;">{{ message }}</p> <br> <textarea v-model="message" placeholder="add multiple lines"></textarea> - checkbox
<input type="checkbox" id="checkbox" v-model="checked"> <label for="checkbox">{{ checked }}</label> <!-- `toggle` is either true or false --> <input type="checkbox" v-model="toggle"> // multiple checkboxes <div id='example-3'> <input type="checkbox" id="jack" value="Jack" v-model="checkedNames"> <label for="jack">Jack</label> <input type="checkbox" id="john" value="John" v-model="checkedNames"> <label for="john">John</label> <input type="checkbox" id="mike" value="Mike" v-model="checkedNames"> <label for="mike">Mike</label> <br> <span>Checked names: {{ checkedNames }}</span> </div> new Vue({ el: '#example-3', data: { checkedNames: [] } }) - radio
<input type="radio" id="one" value="One" v-model="picked"> <label for="one">One</label> <br> <input type="radio" id="two" value="Two" v-model="picked"> <label for="two">Two</label> <br> <span>Picked: {{ picked }}</span> <!-- `picked` is a string "a" when checked --> <input type="radio" v-model="picked" value="a"> - select
<select v-model="selected"> <option disabled value="">Please select one</option> <option>A</option> <option>B</option> <option>C</option> </select> <span>Selected: {{ selected }}</span> new Vue({ el: '...', data: { selected: '' } }) <!-- `selected` is a string "abc" when selected --> <select v-model="selected"> <option value="abc">ABC</option> </select> // select multiple <select v-model="selected" multiple> <option>A</option> <option>B</option> <option>C</option> </select> <br> <span>Selected: {{ selected }}</span> // select v-for <select v-model="selected"> <option v-for="option in options" v-bind:value="option.value"> {{ option.text }} </option> </select> <span>Selected: {{ selected }}</span> new Vue({ el: '...', data: { selected: 'A', options: [ { text: 'One', value: 'A' }, { text: 'Two', value: 'B' }, { text: 'Three', value: 'C' } ] } }) - Use v-bind with v-model to prepopulate
<input type="checkbox" v-model="toggle" v-bind:true-value="a" v-bind:false-value="b" > // when checked: vm.toggle === vm.a // when unchecked: vm.toggle === vm.b <input type="radio" v-model="pick" v-bind:value="a"> // when checked: vm.pick === vm.a <select v-model="selected"> <!-- inline object literal --> <option v-bind:value="{ number: 123 }">123</option> </select> - v-model modifiers
.lazy <!-- synced after "change" instead of "input" --> <input v-model.lazy="msg" > .number :: typecast as a number <input v-model.number="age" type="number"> .trim <input v-model.trim="msg">
- Event in v-model
9.1.6. Render Functions & JSX
9.1.6.1. Render function
To avoid duplicating <slot> in string template
<anchored-heading :level="1">Hello world!</anchored-heading> <script type="text/x-template" id="anchored-heading-template"> <h1 v-if="level === 1"> <slot></slot> </h1> <h2 v-else-if="level === 2"> <slot></slot> </h2> <h3 v-else-if="level === 3"> <slot></slot> </h3> <h4 v-else-if="level === 4"> <slot></slot> </h4> <h5 v-else-if="level === 5"> <slot></slot> </h5> <h6 v-else-if="level === 6"> <slot></slot> </h6> </script> <!-- Vue.component('anchored-heading', { template: '#anchored-heading-template', props: { level: { type: Number, required: true } } }) -->
Use render function (reactive)
Vue.component('anchored-heading', { render: function (createElement) { return createElement( 'h' + this.level, // tag name this.$slots.default // array of children ) }, props: { level: { type: Number, required: true } } })
- createElement Arguments
// @returns {VNode} createElement( // {String | Object | Function} // An HTML tag name, component options, or function // returning one of these. Required. 'div', // {Object} // A data object corresponding to the attributes // you would use in a template. Optional. { // (see details in the next section below) }, // {String | Array} // Children VNodes, built using `createElement()`, // or using strings to get 'text VNodes'. Optional. [ 'Some text comes first.', createElement('h1', 'A headline'), createElement(MyComponent, { props: { someProp: 'foobar' } }) ] ){ // Same API as `v-bind:class` 'class': { foo: true, bar: false }, // Same API as `v-bind:style` style: { color: 'red', fontSize: '14px' }, // Normal HTML attributes attrs: { id: 'foo' }, // Component props props: { myProp: 'bar' }, // DOM properties domProps: { innerHTML: 'baz' }, // Event handlers are nested under `on`, though // modifiers such as in `v-on:keyup.enter` are not // supported. You'll have to manually check the // keyCode in the handler instead. on: { click: this.clickHandler }, // For components only. Allows you to listen to // native events, rather than events emitted from // the component using `vm.$emit`. nativeOn: { click: this.nativeClickHandler }, // Custom directives. Note that the binding's // oldValue cannot be set, as Vue keeps track // of it for you. directives: [ { name: 'my-custom-directive', value: '2', expression: '1 + 1', arg: 'foo', modifiers: { bar: true } } ], // Scoped slots in the form of // { name: props => VNode | Array<VNode> } scopedSlots: { default: props => createElement('span', props.text) }, // The name of the slot, if this component is the // child of another component slot: 'name-of-slot', // Other special top-level properties key: 'myKey', ref: 'myRef' } - createElement Constraints
All VNodes in the component tree must be unique. This is invalid
render: function (createElement) { var myParagraphVNode = createElement('p', 'hi') return createElement('div', [ // Yikes - duplicate VNodes! myParagraphVNode, myParagraphVNode ]) }Use factory function
render: function (createElement) { return createElement('div', Array.apply(null, { length: 20 }).map(function () { return createElement('p', 'hi') }) ) } - render for v-if and v-for
- render for v-model
- redner for event & key modifiers
- render for <slot>
Static slot contents as Arrays of VNodes from this.$slots
render: function (createElement) { // `<div><slot></slot></div>` return createElement('div', this.$slots.default) }Scoped slot
render: function (createElement) { // `<div><slot :text="msg"></slot></div>` return createElement('div', [ this.$scopedSlots.default({ text: this.msg }) ]) } // pass scoped slots to a child component render (createElement) { return createElement('div', [ createElement('child', { // pass `scopedSlots` in the data object // in the form of { name: props => VNode | Array<VNode> } scopedSlots: { default: function (props) { return createElement('span', props.text) } } }) ]) }
9.1.6.2. JSX
9.1.7. vm.computed property, vm.watch property
9.1.7.1. Computed property
Example
<div id="example"> <p>Original message: "{{ message }}"</p> <p>Computed reversed message: "{{ reversedMessage }}"</p> <!-- same as <p>Reversed message: "{{ reverseMessage() }}"</p> --> </div>
var vm = new Vue({ el: '#example', data: { message: 'Hello' }, computed: { // a computed getter reversedMessage: function () { // `this` points to the vm instance return this.message.split('').reverse().join('') } } /* same as method methods: { reverseMessage: function () { return this.message.split('').reverse().join('') } '} */ })
vm.computed.reversedMessageis reactive tovm.$data.messageand it's cached when there is no changeWhile
methodwill run every time the view updates (re-render). Hence, never define computed property without refering tovm.$datae.g. Don't do thiscomputed: { now: function () { return Date.now() } }Computed Setter and Getter
computed: { fullName: { // getter get: function () { return this.firstName + ' ' + this.lastName }, // setter set: function (newValue) { var names = newValue.split(' ') this.firstName = names[0] this.lastName = names[names.length - 1] } } }
9.1.7.2. vm.watch property - Watchers
vm.watchis to watch a propertyvm.data.propAchangeTry to avoid using watch too much, use computed property instead Instead of
var vm = new Vue({ el: '#demo', data: { firstName: 'Foo', lastName: 'Bar', fullName: 'Foo Bar' }, watch: { firstName: function (newval, oldval) { this.fullName = newval + ' ' + this.lastName }, lastName: function (newval, oldval) { this.fullName = this.firstName + ' ' + newval } } })
Use
var vm = new Vue({ el: '#demo', data: { firstName: 'Foo', lastName: 'Bar' }, computed: { fullName: function () { return this.firstName + ' ' + this.lastName } } })
Another example
<div id="watch-example"> <p> Ask a yes/no question: <input v-model="question"> </p> <p>{{ answer }}</p> </div> <!-- Since there is already a rich ecosystem of ajax libraries --> <!-- and collections of general-purpose utility methods, Vue core --> <!-- is able to remain small by not reinventing them. This also --> <!-- gives you the freedom to use what you're familiar with. --> <script src="https://cdn.jsdelivr.net/npm/axios@0.12.0/dist/axios.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/lodash@4.13.1/lodash.min.js"></script>
var watchExampleVM = new Vue({ el: '#watch-example', data: { question: '', answer: 'I cannot give you an answer until you ask a question!' }, watch: { // whenever question changes, this function will run question: function (newQuestion) { this.answer = 'Waiting for you to stop typing...' this.getAnswer() } }, methods: { // _.debounce is a function provided by lodash to limit how // often a particularly expensive operation can be run. // In this case, we want to limit how often we access // yesno.wtf/api, waiting until the user has completely // finished typing before making the ajax request. To learn // more about the _.debounce function (and its cousin // _.throttle), visit: https://lodash.com/docs#debounce getAnswer: _.debounce( function () { if (this.question.indexOf('?') === -1) { this.answer = 'Questions usually contain a question mark. ;-)' return } this.answer = 'Thinking...' var vm = this axios.get('https://yesno.wtf/api') .then(function (response) { vm.answer = _.capitalize(response.data.answer) }) .catch(function (error) { vm.answer = 'Error! Could not reach the API. ' + error }) }, // This is the number of milliseconds we wait for the // user to stop typing. 500 ) } })
9.1.8. Transition & Animation
9.1.8.1. <transition> wrapper component
- Wrap v-if, v-show, dynamic components and component root nodes
- CSS Transition
<div id="demo"> <button v-on:click="show = !show"> Toggle </button> <transition name="fade"> <p v-if="show">hello</p> </transition> </div>
new Vue({ el: '#demo', data: { show: true } })
.fade-enter-active, .fade-leave-active { transition: opacity .5s } .fade-enter, .fade-leave-to /* .fade-leave-active below version 2.1.8 */ { opacity: 0 } /* more examples using pure css * Enter and leave animations can use different * durations and timing functions. */ .slide-fade-enter-active { transition: all .3s ease; } .slide-fade-leave-active { transition: all .8s cubic-bezier(1.0, 0.5, 0.8, 1.0); } .slide-fade-enter, .slide-fade-leave-to /* .slide-fade-leave-active below version 2.1.8 */ { transform: translateX(10px); opacity: 0; }
- Classes
*-enter- before element inserted > one frame after element is inserted
*-enter-active- before element is inserted > removed when transition/animation finishes. Use it to define during, delay and easing curve for the entering transition.
- (no term)
*-enter-to: one frame after element is inserted > transition/animation finishes.*-leave- leave transition is triggered > after one frame
*-leave-active- leave transition is triggered > transition/animation finishes. Use it to define duration, delay and easing curve for the leaving transition.
*-leave-to- one frame after a leave transition is triggered > transition/animation finishes.
- (no term)
*-enteris enter start,*-enter-tois enter end- (no term)
*-leaveis leave start,*-leave-tois leave end- (no term)
v-prefix is used when <transition> has no name
- Classes
- CSS Animations
<div id="example-2"> <button @click="show = !show">Toggle show</button> <transition name="bounce"> <p v-if="show">Look at me!</p> </transition> </div> new Vue({ el: '#example-2', data: { show: true } }) .bounce-enter-active { animation: bounce-in .5s; } .bounce-leave-active { animation: bounce-in .5s reverse; } @keyframes bounce-in { 0% { transform: scale(0); } 50% { transform: scale(1.5); } 100% { transform: scale(1); } } - Custom Transition Class
Override the default class names such as Animate.css
<link href="https://cdn.jsdelivr.net/npm/animate.css@3.5.1" rel="stylesheet" type="text/css"> <div id="example-3"> <button @click="show = !show"> Toggle render </button> <transition name="custom-classes-transition" enter-active-class="animated tada" leave-active-class="animated bounceOutRight" > <p v-if="show">hello</p> </transition> </div>
- prop
<transition :duration="1000">...</transition> <transition :duration="{ enter: 500, leave: 800 }">...</transition> - event hooks
When using JavaScript-only transitions, the done callbacks are required for the enter and leave hooks.
- events
<transition v-on:before-enter="beforeEnter" v-on:enter="enter" v-on:after-enter="afterEnter" v-on:enter-cancelled="enterCancelled" v-on:before-leave="beforeLeave" v-on:leave="leave" v-on:after-leave="afterLeave" v-on:leave-cancelled="leaveCancelled" > <!-- ... --> </transition>
- methods
// ... methods: { // -------- // ENTERING // -------- beforeEnter: function (el) { // ... }, // the done callback is optional when // used in combination with CSS enter: function (el, done) { // ... done() }, afterEnter: function (el) { // ... }, enterCancelled: function (el) { // ... }, // -------- // LEAVING // -------- beforeLeave: function (el) { // ... }, // the done callback is optional when // used in combination with CSS leave: function (el, done) { // ... done() }, afterLeave: function (el) { // ... }, // leaveCancelled only available with v-show leaveCancelled: function (el) { // ... } }When javascript-only transistions, better to add v-bind:css="false"
- Example using Velocity.js
<!-- Velocity works very much like jQuery.animate and is a great option for JavaScript animations --> <script src="https://cdnjs.cloudflare.com/ajax/libs/velocity/1.2.3/velocity.min.js"></script> <div id="example-4"> <button @click="show = !show"> Toggle </button> <transition v-on:before-enter="beforeEnter" v-on:enter="enter" v-on:leave="leave" v-bind:css="false" > <p v-if="show"> Demo </p> </transition> </div> new Vue({ el: '#example-4', data: { show: false }, methods: { beforeEnter: function (el) { el.style.opacity = 0 }, enter: function (el, done) { Velocity(el, { opacity: 1, fontSize: '1.4em' }, { duration: 300 }) Velocity(el, { fontSize: '1em' }, { complete: done }) }, leave: function (el, done) { Velocity(el, { translateX: '15px', rotateZ: '50deg' }, { duration: 600 }) Velocity(el, { rotateZ: '100deg' }, { loop: 2 }) Velocity(el, { rotateZ: '45deg', translateY: '30px', translateX: '30px', opacity: 0 }, { complete: done }) } } })
- events
- Transition on Initial Render
Use appear attribute
// default only uses enter and leave <transition appear> <!-- ... --> </transition> // specify more or custom CSS classes <transition appear appear-class="custom-appear-class" appear-to-class="custom-appear-to-class" (2.1.8+) appear-active-class="custom-appear-active-class" > <!-- ... --> </transition> // and custom javascript hooks <transition appear v-on:before-appear="customBeforeAppearHook" v-on:appear="customAppearHook" v-on:after-appear="customAfterAppearHook" v-on:appear-cancelled="customAppearCancelledHook" > <!-- ... --> </transition>
- Transition between elements
<transition> <button v-if="docState === 'saved'" key="saved"> Edit </button> <button v-if="docState === 'edited'" key="edited"> Save </button> <button v-if="docState === 'editing'" key="editing"> Cancel </button> </transition> // It's equivalent to <transition> <button v-bind:key="docState"> {{ buttonMessage }} </button> </transition> // ... computed: { buttonMessage: function () { switch (this.docState) { case 'saved': return 'Edit' case 'edited': return 'Save' case 'editing': return 'Cancel' } } } - Transition modes
Default behavior is entering and leaving happens simultaneously.
<transition name="fade" mode="out-in"> <!-- ... the buttons ... --> </transition>
in-out :: new element transitions in first then when complete, the current element transitions out. out-in :: current element transitions out first then when complete, the new element transitions in.
- Transition between dynamic components
key attribute is not required
<transition name="component-fade" mode="out-in"> <component v-bind:is="view"></component> </transition> new Vue({ el: '#transition-components-demo', data: { view: 'v-a' }, components: { 'v-a': { template: '<div>Component A</div>' }, 'v-b': { template: '<div>Component B</div>' } } })
9.1.8.2. Loop items transition transition-group
- Attributes
tag="span"- Default is to animate each
spanchild item inside the<transition-group>, this example change it to animatepelements key- required
- Animate entering and leaving
<div id="list-demo"> <button v-on:click="add">Add</button> <button v-on:click="remove">Remove</button> <transition-group name="list" tag="p"> <span v-for="item in items" v-bind:key="item" class="list-item"> {{ item }} </span> </transition-group> </div>
new Vue({ el: '#list-demo', data: { items: [1,2,3,4,5,6,7,8,9], nextNum: 10 }, methods: { randomIndex: function () { return Math.floor(Math.random() * this.items.length) }, add: function () { this.items.splice(this.randomIndex(), 0, this.nextNum++) }, remove: function () { this.items.splice(this.randomIndex(), 1) }, } })
.list-item { display: inline-block; margin-right: 10px; } .list-enter-active, .list-leave-active { transition: all 1s; } .list-enter, .list-leave-to /* .list-leave-active below version 2.1.8 */ { opacity: 0; transform: translateY(30px); }
- Animate when order is changed,
v-move,*-moveclass
- Vue uses FLIP and FLIP doesn't work with elements set to
display:inline. Set it todisplay: inline-blockor place elements in a flex context
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.14.1/lodash.min.js"></script> <div id="flip-list-demo" class="demo"> <button v-on:click="shuffle">Shuffle</button> <transition-group name="flip-list" tag="ul"> <li v-for="item in items" v-bind:key="item"> {{ item }} </li> </transition-group> </div>
new Vue({ el: '#flip-list-demo', data: { items: [1,2,3,4,5,6,7,8,9] }, methods: { shuffle: function () { this.items = _.shuffle(this.items) } } })
.flip-list-move { transition: transform 1s; }
- Vue uses FLIP and FLIP doesn't work with elements set to
- JavaScript event hooks
- https://vuejs.org/v2/guide/transitions.html#JavaScript-Hooks
- By assigning different delay, list items can be transitioned in/out in the same order as they first appeared
<script src="https://cdnjs.cloudflare.com/ajax/libs/velocity/1.2.3/velocity.min.js"></script> <div id="staggered-list-demo"> <input v-model="query"> <transition-group name="staggered-fade" tag="ul" v-bind:css="false" v-on:before-enter="beforeEnter" v-on:enter="enter" v-on:leave="leave" > <li v-for="(item, index) in computedList" v-bind:key="item.msg" v-bind:data-index="index" >{{ item.msg }}</li> </transition-group> </div>
new Vue({ el: '#staggered-list-demo', data: { query: '', list: [ { msg: 'Bruce Lee' }, { msg: 'Jackie Chan' }, { msg: 'Chuck Norris' }, { msg: 'Jet Li' }, { msg: 'Kung Fury' } ] }, computed: { computedList: function () { var vm = this return this.list.filter(function (item) { return item.msg.toLowerCase().indexOf(vm.query.toLowerCase()) !== -1 }) } }, methods: { beforeEnter: function (el) { el.style.opacity = 0 el.style.height = 0 }, enter: function (el, done) { var delay = el.dataset.index * 150 setTimeout(function () { Velocity( el, { opacity: 1, height: '1.6em' }, { complete: done } ) }, delay) }, leave: function (el, done) { var delay = el.dataset.index * 150 setTimeout(function () { Velocity( el, { opacity: 0, height: 0 }, { complete: done } ) }, delay) } } })
9.1.8.3. Make transition reusable
Vue.component('my-special-transition', {
template: '\
<transition\
name="very-special-transition"\
mode="out-in"\
v-on:before-enter="beforeEnter"\
v-on:after-enter="afterEnter"\
>\
<slot></slot>\
</transition>\
',
methods: {
beforeEnter: function (el) {
// ...
},
afterEnter: function (el) {
// ...
}
}
})
Functional component
Vue.component('my-special-transition', {
functional: true,
render: function (createElement, context) {
var data = {
props: {
name: 'very-special-transition',
mode: 'out-in'
},
on: {
beforeEnter: function (el) {
// ...
},
afterEnter: function (el) {
// ...
}
}
}
return createElement('transition', data, context.children)
}
})
9.1.8.4. State Transition
Achieve transition by changing state (e.g. prop) of data.
- Watchers, Tween.js, Color.js
<script src="https://cdn.jsdelivr.net/npm/tween.js@16.3.4"></script> <div id="animated-number-demo"> <input v-model.number="number" type="number" step="20"> <p>{{ animatedNumber }}</p> </div> new Vue({ el: '#animated-number-demo', data: { number: 0, animatedNumber: 0 }, watch: { number: function(newValue, oldValue) { var vm = this function animate () { if (TWEEN.update()) { requestAnimationFrame(animate) } } new TWEEN.Tween({ tweeningNumber: oldValue }) .easing(TWEEN.Easing.Quadratic.Out) .to({ tweeningNumber: newValue }, 500) .onUpdate(function () { vm.animatedNumber = this.tweeningNumber.toFixed(0) }) .start() animate() } } })
9.1.9. Mixins
9.1.9.1. Local vs Global
// define a mixin object var myMixin = { created: function () { this.hello() }, methods: { hello: function () { console.log('hello from mixin!') } } } // define a component that uses this mixin var Component = Vue.extend({ mixins: [myMixin] }) var component = new Component() // => "hello from mixin!"
- Global mixin
Use global mixins sparsely and carefully, because it affects every single Vue instance created, including third party components. In most cases, you should only use it for custom option handling like demonstrated in the example above. It’s also a good idea to ship them as Plugins to avoid duplicate application
// inject a handler for `myOption` custom option Vue.mixin({ created: function () { var myOption = this.$options.myOption if (myOption) { console.log(myOption) } } }) new Vue({ myOption: 'hello!' }) // => "hello!"
9.1.9.2. Merge option with hook functions
mixin will be called before the component's own hooks.
var mixin = {
created: function () {
console.log('mixin hook called')
}
}
new Vue({
mixins: [mixin],
created: function () {
console.log('component hook called')
}
})
// => "mixin hook called"
// => "component hook called"
9.1.9.3. Merge option with Object value
The component's options will take priority when conflict keys exist
var mixin = {
methods: {
foo: function () {
console.log('foo')
},
conflicting: function () {
console.log('from mixin')
}
}
}
var vm = new Vue({
mixins: [mixin],
methods: {
bar: function () {
console.log('bar')
},
conflicting: function () {
console.log('from self')
}
}
})
vm.foo() // => "foo"
vm.bar() // => "bar"
vm.conflicting() // => "from self"
9.1.9.4. Custom Option Merge Strategies
Vue.config.optionMergeStrategies.myOption = function (toVal, fromVal) {
// return mergedVal
}
var strategies = Vue.config.optionMergeStrategies
strategies.myOption = strategies.methods // the default strategy for most object-based options
// more advanced
const merge = Vue.config.optionMergeStrategies.computed
Vue.config.optionMergeStrategies.vuex = function (toVal, fromVal) {
if (!toVal) return fromVal
if (!fromVal) return toVal
return {
getters: merge(toVal.getters, fromVal.getters),
state: merge(toVal.state, fromVal.state),
actions: merge(toVal.actions, fromVal.actions)
}
}
9.1.10. Custom Directive
9.1.10.1. Basics
<input v-focus>
// Register a global custom directive called v-focus
Vue.directive('focus', {
// When the bound element is inserted into the DOM...
inserted: function (el) {
// Focus the element
el.focus()
}
})
Register it locally for a component, using the directives option inside the component
directives: {
focus: {
// directive definition
inserted: function (el) {
el.focus()
}
}
}
9.1.10.2. Option - Hook Functions
Other than inserted, there're other optional hooks
- bind
- called only once, when the directive is first bound to the element. This is where you can do one-time setup work.
- inserted
- called when the bound element has been inserted into its parent node (this only guarantees parent node presence, not necessarily in-document).
- update
- called after the containing component’s VNode has updated, but possibly before its children have updated. The directive’s value may or may not have changed, but you can skip unnecessary updates by comparing the binding’s current and old values (see below on hook arguments).
- componentUpdated
- called after the containing component’s VNode and the VNodes of its children have updated.
unbind: called only once, when the directive is unbound from the element.
Shorthand for same behavior on bind and update
Vue.directive('color-swatch', function (el, binding) {
el.style.backgroundColor = binding.value
})
- Hooks are passed these arguments
Apart from el, you should treat these arguments as read-only and never modify them. If you need to share information across hooks, it is recommended to do so through element’s dataset.
- el
- The element the directive is bound to. This can be used to directly manipulate the DOM.
- binding
- An object containing the following properties.
- name
- The name of the directive, without the v- prefix.
- value
- The value passed to the directive. For example in v-my-directive="1 + 1", the value would be 2.
- oldValue
- The previous value, only available in update and componentUpdated. It is available whether or not the value has changed.
- expression
- The expression of the binding as a string. For example in v-my-directive="1 + 1", the expression would be "1 + 1".
- arg
- The argument passed to the directive, if any. For example in v-my-directive:foo, the arg would be "foo".
- modifiers
- An object containing modifiers, if any. For example in v-my-directive.foo.bar, the modifiers object would be
{ foo: true, bar: true }
- vnode
- The virtual node produced by Vue’s compiler. See the VNode API for full details.
- oldVnode
- The previous virtual node, only available in the update and componentUpdated hooks.
9.1.10.3. Example
<div id="hook-arguments-example" v-demo:foo.a.b="message"></div>
Vue.directive('demo', {
bind: function (el, binding, vnode) {
var s = JSON.stringify
el.innerHTML =
'name: ' + s(binding.name) + '<br>' +
'value: ' + s(binding.value) + '<br>' +
'expression: ' + s(binding.expression) + '<br>' +
'argument: ' + s(binding.arg) + '<br>' +
'modifiers: ' + s(binding.modifiers) + '<br>' +
'vnode keys: ' + Object.keys(vnode).join(', ')
}
})
new Vue({
el: '#hook-arguments-example',
data: {
message: 'hello!'
}
})
// result
name: "demo"
value: "hello!"
expression: "message"
argument: "foo"
modifiers: {"a":true,"b":true}
vnode keys: tag, data, children, text, elm, ns, context, functionalContext, functionalOptions, functionalScopeId, key, componentOptions, componentInstance, parent, raw, isStatic, isRootInsert, isComment, isCloned, isOnce, asyncFactory, asyncMeta, isAsyncPlaceholder
Pass multiple values to directive
<div v-demo="{ color: 'white', text: 'hello!' }"></div>
Vue.directive('demo', function (el, binding) {
console.log(binding.value.color) // => "white"
console.log(binding.value.text) // => "hello!"
})
9.1.11. Plugin vue:plugin
Third party plugins and libraries https://github.com/vuejs/awesome-vue#components--libraries
9.1.11.1. Basics
Plugins usually add global-level functionality to Vue. There is no strictly defined scope for a plugin - there are typically several types of plugins you can write:
- Add some global methods or properties. e.g. vue-custom-element
- Add one or more global assets: directives/filters/transitions etc. e.g. vue-touch
- Add some component options by global mixin. e.g. vue-router
- Add some Vue instance methods by attaching them to Vue.prototype.
- A library that provides an API of its own, while at the same time injecting some combination of the above. e.g. vue-router
Official Vue.js plugins such as vue-router automatically calls Vue.use() if vue is available as a global variable.
MyPlugin.install = function (Vue, options) {
// 1. add global method or property
Vue.myGlobalMethod = function () {
// something logic ...
}
// 2. add a global asset
Vue.directive('my-directive', {
bind (el, binding, vnode, oldVnode) {
// something logic ...
}
...
})
// 3. inject some component options
Vue.mixin({
created: function () {
// something logic ...
}
...
})
// 4. add an instance method
Vue.prototype.$myMethod = function (methodOptions) {
// something logic ...
}
}
Use a Plugin
// calls `MyPlugin.install(Vue)`
Vue.use(MyPlugin)
Vue.use(MyPlugin, { someOption: true })
// When using CommonJS via Browserify or Webpack
var Vue = require('vue')
var VueRouter = require('vue-router')
// Don't forget to call this
Vue.use(VueRouter)
9.1.11.2. vuex Namespace in Plugin
9.1.12. Production Development
Without build tools, directly inject javascript Prod :: https://cdn.jsdelivr.net/npm/vue Dev :: https://unpkg.com/vue
9.1.13. vue-router vue:plugin:router vue:router
VueRouter:option:x => option:x
9.1.13.1. <router-view>, <router-link>, option:routes, this.$router, this.$route
<script src="https://unpkg.com/vue/dist/vue.js"></script>
<script src="https://unpkg.com/vue-router/dist/vue-router.js"></script>
<div id="app">
<h1>Hello App!</h1>
<p>
<!-- use router-link component for navigation. -->
<!-- specify the link by passing the `to` prop. -->
<!-- `<router-link>` will be rendered as an `<a>` tag by default -->
<!-- it gets .router-link-active class when the route is matched -->
<router-link to="/foo">Go to Foo</router-link>
<router-link to="/bar">Go to Bar</router-link>
</p>
<!-- route outlet -->
<!-- component matched by the route will render here -->
<router-view></router-view>
</div>
// 0. If using a module system (e.g. via vue-cli), import Vue and VueRouter
// and then call `Vue.use(VueRouter)`.
// 1. Define route components.
// These can be imported from other files
const Foo = { template: '<div>foo</div>' }
const Bar = { template: '<div>bar</div>' }
// 2. Define some routes
// Each route should map to a component. The "component" can
// either be an actual component constructor created via
// `Vue.extend()`, or just a component options object.
// We'll talk about nested routes later.
const routes = [
{ path: '/foo', component: Foo },
{ path: '/bar', component: Bar }
]
// 3. Create the router instance and pass the `routes` option
// You can pass in additional options here, but let's
// keep it simple for now.
const router = new VueRouter({
routes // short for `routes: routes`
})
// 4. Create and mount the root instance.
// Make sure to inject the router with the router option to make the
// whole app router-aware.
const app = new Vue({
router
}).$mount('#app')
// Now the app has started!
In any component, this.$router vue:router:router and this.$route vue:router:route are available
// Home.vue
export default {
computed: {
username () {
// We will see what `params` is shortly
return this.$route.params.username
}
},
methods: {
goBack () {
window.history.length > 1
? this.$router.go(-1)
: this.$router.push('/')
}
}
}
9.1.13.2. option:routes, $route.params, regex, option:routes[]:name, option:routes[]:component
const User = {
template: '<div>User {{ $route.params.id }}</div>'
}
const router = new VueRouter({
routes: [
// dynamic segments start with a colon
{ path: '/user/:id', component: User }
]
})
- regex pattern in option:routes
vue-router uses path-to-regexp whihc is used by Express as well. The earlier a route is defined, the higher priority it gets.
import Vue from 'vue' import VueRouter from 'vue-router' Vue.use(VueRouter) // The matching uses path-to-regexp, which is the matching engine used // by express as well, so the same matching rules apply. // For detailed rules, see https://github.com/pillarjs/path-to-regexp const router = new VueRouter({ mode: 'history', base: __dirname, routes: [ { path: '/' }, // params are denoted with a colon ":" { path: '/params/:foo/:bar' }, // a param can be made optional by adding "?" { path: '/optional-params/:foo?' }, // a param can be followed by a regex pattern in parens // this route will only be matched if :id is all numbers { path: '/params-with-regex/:id(\\d+)' }, // asterisk can match anything { path: '/asterisk/*' }, // make part of the path optional by wrapping with parens and add "?" { path: '/optional-group/(foo/)?bar' } ] }) new Vue({ router, template: ` <div id="app"> <h1>Route Matching</h1> <ul> <li><router-link to="/">/</router-link></li> <li><router-link to="/params/foo/bar">/params/foo/bar</router-link></li> <li><router-link to="/optional-params">/optional-params</router-link></li> <li><router-link to="/optional-params/foo">/optional-params/foo</router-link></li> <li><router-link to="/params-with-regex/123">/params-with-regex/123</router-link></li> <li><router-link to="/params-with-regex/abc">/params-with-regex/abc</router-link></li> <li><router-link to="/asterisk/foo">/asterisk/foo</router-link></li> <li><router-link to="/asterisk/foo/bar">/asterisk/foo/bar</router-link></li> <li><router-link to="/optional-group/bar">/optional-group/bar</router-link></li> <li><router-link to="/optional-group/foo/bar">/optional-group/foo/bar</router-link></li> </ul> <p>Route context</p> <pre>{{ JSON.stringify($route, null, 2) }}</pre> </div> ` }).$mount('#app') - Named routes, option:routes[]:name
const router = new VueRouter({ routes: [ { path: '/user/:userId', name: 'user', component: User } ] }) <router-link :to="{ name: 'user', params: { userId: 123 }}">User</router-link> router.push({ name: 'user', params: { userId: 123 }})
9.1.13.3. option:routes[]:components, multiple <router-view>'s
<router-view class="view one"></router-view>
<router-view class="view two" name="a"></router-view>
<router-view class="view three" name="b"></router-view>
const router = new VueRouter({
routes: [
{
path: '/',
components: {
default: Foo,
a: Bar,
b: Baz
}
}
]
})
9.1.13.4. option:routes[]:redirect, option:routes[]:alias
A redirect means when the user visits /a, and URL will be replaced by /b, and then matched as /b.
const router = new VueRouter({
routes: [
{ path: '/a', redirect: '/b' }
]
})
const router = new VueRouter({
routes: [
{ path: '/a', redirect: { name: 'foo' }}
]
})
const router = new VueRouter({
routes: [
{ path: '/a', redirect: to => {
// the function receives the target route as the argument
// return redirect path/location here.
}}
]
})
An alias of /a as /b means when the user visits /b, the URL remains /b, but it will be matched as if the user is visiting /a.
const router = new VueRouter({
routes: [
{ path: '/a', component: A, alias: '/b' }
]
})
9.1.13.5. Route object, $route, this.$route, option:scrollBehavior vue:router:route
- https://router.vuejs.org/en/api/route-object.html
$route => this.$route, router.match(location)router.beforeEach((to, from, next) => { // `to` and `from` are both route objects }) const router = new VueRouter({ scrollBehavior (to, from, savedPosition) { // `to` and `from` are both route objects } })- absolute path e.g. /foo/bar
- {} or key/value pairs of dynamic segments and star segments
- {} or key/vale pairs of the query string /foo?user=1 $route.query.user == 1
- empty string or anything with #
- string query and hash
- e.g. /foo/bar, then .matched is an array of objects in parent to child order.
9.1.13.6. option:scrollBehavior
const router = new VueRouter({
routes: [...],
scrollBehavior (to, from, savedPosition) {
// return desired position
}
})
savedPosition, is only available if this is a popstate navigation (triggered by the browser's back/forward buttons).
Should return either one of these
- { x: number, y: number }
- { selector: string, offset? : { x: number, y: number }}
Scroll to top for all route navigations
scrollBehavior (to, from, savedPosition) {
return { x: 0, y: 0 }
}
scrollBehavior (to, from, savedPosition) {
if (savedPosition) {
return savedPosition
} else {
return { x: 0, y: 0 }
}
}
// scroll to anchor
scrollBehavior (to, from, savedPosition) {
if (to.hash) {
return {
selector: to.hash
// , offset: { x: 0, y: 10 }
}
}
}
9.1.13.7. option:routes[]:props, component:props
Decouple $route in component. Before
const User = {
template: '<div>User {{ $route.params.id }}</div>'
}
const router = new VueRouter({
routes: [
{ path: '/user/:id', component: User }
]
})
Now
const User = {
props: ['id'],
template: '<div>User {{ id }}</div>'
}
const router = new VueRouter({
routes: [
{ path: '/user/:id', component: User, props: true }
// for routes with named views, you have to define the `props` option for each named view:
{
path: '/user/:id',
components: { default: User, sidebar: Sidebar },
props: { default: true, sidebar: false }
}
]
})
option:routes[]:props is set to true, the route.params will be set as the component props.
props is an object
const router = new VueRouter({
routes: [
{ path: '/hello/:name', component: Hello, props: true }, // Pass route.params to props
{ path: '/promotion/from-newsletter', component: Promotion, props: { newsletterPopup: false } },
{ path: '/static', component: Hello, props: { name: 'world' }}, // static values
]
})
props is a function The URL /search?q=vue would pass {query: 'vue'} as props to the SearchUser component.
function dynamicPropsFn (route) {
const now = new Date()
return {
name: (now.getFullYear() + parseInt(route.params.years)) + '!'
}
}
const router = new VueRouter({
routes: [
{ path: '/search', component: SearchUser, props: (route) => ({ query: route.query.q }) }
{ path: '/dynamic/:years', component: Hello, props: dynamicPropsFn }, // custom logic for mapping between route and props
]
})
9.1.13.8. React to params changes, component:watch, component:beforeRouteUpdate
vue:router:react vue:router:component:beforeRouteUpdate vue:router:component:watch params or query chnages won't trigger enter/leave navigation guards and hence this is needed to react to changes.
const User = {
template: '...',
watch: {
'$route' (to, from) {
// react to route changes...
}
}
}
const User = {
template: '...',
beforeRouteUpdate (to, from, next) {
// react to route changes...
// don't forget to call next()
}
}
9.1.13.9. Nested routes, <router-view>, option:routes:children
/user/foo/profile (component:user and component:profile) and /user/foo/posts (component:user and component:posts)
<div id="app">
<router-view></router-view>
</div>
const User = {
template: `
<div>User {{ $route.params.id }}</div>
<router-view></router-view>
`
}
const router = new VueRouter({
routes: [
{ path: '/user/:id', component: User,
children: [
// UserHome will be rendered inside User's <router-view>
// when /user/:id is matched
{ path: '', component: UserHome },
{
// UserProfile will be rendered inside User's <router-view>
// when /user/:id/profile is matched
path: 'profile',
component: UserProfile
},
{
// UserPosts will be rendered inside User's <router-view>
// when /user/:id/posts is matched
path: 'posts',
component: UserPosts
}
]
}
]
})
- option:routes:children[]:meta
const router = new VueRouter({ routes: [ { path: '/foo', component: Foo, children: [ { path: 'bar', component: Bar, // a meta field meta: { requiresAuth: true } } ] } ] })router.beforeEach((to, from, next) => { if (to.matched.some(record => record.meta.requiresAuth)) { // this route requires auth, check if logged in // if not, redirect to login page. if (!auth.loggedIn()) { next({ path: '/login', query: { redirect: to.fullPath } }) } else { next() } } else { next() // make sure to always call next()! } })
9.1.13.10. Programmatic Navigation, this.$router, router.*
vue:router:router vue:router:router.push
Both router and this.$router can be used in component. this.$router means vue-router is imported in every component.
<router-link :to="..."> is equivalent to router.push(...)
params are ignored if a path is provided.
// literal string path
router.push('home')
// object
router.push({ path: 'home' })
// named route
router.push({ name: 'user', params: { userId: 123 }})
// with query, resulting in /register?plan=private
router.push({ path: 'register', query: { plan: 'private' }})
const userId = 123
router.push({ name: 'user', params: { userId }}) // -> /user/123
router.push({ path: `/user/${userId}` }) // -> /user/123
// This will NOT work
router.push({ path: '/user', params: { userId }}) // -> /user
- router.push, router.replace, router.push.onComplete, router.push.onAbort
Callbacks as the 2nd and 3rd arguments. onComplete :: after all async hooks are resolved onAbort :: navigated to the same route, or to a different route before current navigation has finished.
If destination is the same as the current route and only params are changing (/users/1 -> /users/2), use beforeRouteUpdate to react. vue:router:react
router.push(location, onComplete?, onAbort?) router.replace(location, onComplete?, onAbort?) :: without pushing a new history entry
- router.go(n)
Similar to window.history.go(n)
// go forward by one record, the same as history.forward() router.go(1) // go back by one record, the same as history.back() router.go(-1) // go forward by 3 records router.go(3) // fails silently if there aren't that many records. router.go(-100) router.go(100)
9.1.13.11. option:mode
Default is hash mode, change to history mode to change URL without a page reload.
const router = new VueRouter({
mode: 'history',
routes: [
// some path
// catch 404
{ path: '*', component: NotFoundComponent }
]
})
Apache
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.html$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.html [L]
</IfModule>
nginx
location / {
try_files $uri $uri/ /index.html;
}
9.1.13.12. Guards
- Navigation triggered.
- Call leave guards in deactivated components.
- Call global beforeEach guards.
- Call beforeRouteUpdate guards in reused components (2.2+).
- Call beforeEnter in route configs.
- Resolve async route components.
- Call beforeRouteEnter in activated components.
- Call global beforeResolve guards (2.5+).
- Navigation confirmed.
- Call global afterEach hooks.
- DOM updates triggered.
- Call callbacks passed to next in beforeRouteEnter guards with instantiated instances.
- global, router.beforeEach, router.beforeResolve, router.afterEach
Guards/hooks may be resolved asynchronously, the navigation is considered pending before all hooks have been resolved (confirmed). vue:router:router.beforeEach
const router = new VueRouter({ ... }) router.beforeEach((to, from, next) => { // ... })Always call next function to resolve.
next() :: move on to the next hook in the pipeline. next(false) :: abort the current navigation. If the URL was changed (either manually by the user or via back button), it will be reset to that of the from route. next('') or next({ path: '' }) :: redirect to a different location. The current navigation will be aborted and a new one will be started. A location object can be passed. Refer to vue:router:router.push next(error) :: navigation will be aborted and the error will be passed to callbacks registered via router.onError().
router.beforeResolve is similar to router.beforeEach with the difference that resolve guards will be called right before the navigation is confirmed, after all in-component guards and async route components are resolved.
router.afterEach do not get a next function and cannot affect the navigation
router.afterEach((to, from) => { // ... }) - per-route, option:routes[]:beforeEnter
Same as vue:router:router.beforeEach
const router = new VueRouter({ routes: [ { path: '/foo', component: Foo, beforeEnter: (to, from, next) => { // ... } } ] }) - in-component, component:beforeRouteEnter, component:beforeRouteUpdate, component:beforeRouteLeave
const Foo = { template: `...`, beforeRouteEnter (to, from, next) { // called before the route that renders this component is confirmed. // does NOT have access to `this` component instance, // because it has not been created yet when this guard is called! // you can access to component instance via `vm` next(vm => { // it will be called when the navigation is confirmed // access to component instance via `vm` }) }, beforeRouteUpdate (to, from, next) { // called when the route that renders this component has changed, // but this component is reused in the new route. // For example, for a route with dynamic params `/foo/:id`, when we // navigate between `/foo/1` and `/foo/2`, the same `Foo` component instance // will be reused, and this hook will be called when that happens. // has access to `this` component instance. }, beforeRouteLeave (to, from, next) { // called when the route that renders this component is about to // be navigated away from. // has access to `this` component instance. const answer = window.confirm('Do you really want to leave? you have unsaved changes!') if (answer) { next() } else { next(false) } } }
9.1.13.13. Fetch data
Fetch after navigation
<template>
<div class="post">
<div class="loading" v-if="loading">
Loading...
</div>
<div v-if="error" class="error">
{{ error }}
</div>
<div v-if="post" class="content">
<h2>{{ post.title }}</h2>
<p>{{ post.body }}</p>
</div>
</div>
</template>
export default {
data () {
return {
loading: false,
post: null,
error: null
}
},
created () {
// fetch the data when the view is created and the data is
// already being observed
this.fetchData()
},
watch: {
// call again the method if the route changes
'$route': 'fetchData'
},
methods: {
fetchData () {
this.error = this.post = null
this.loading = true
// replace `getPost` with your data fetching util / API wrapper
getPost(this.$route.params.id, (err, post) => {
this.loading = false
if (err) {
this.error = err.toString()
} else {
this.post = post
}
})
}
}
}
Fetch before navigation
export default {
data () {
return {
post: null,
error: null
}
},
beforeRouteEnter (to, from, next) {
getPost(to.params.id, (err, post) => {
next(vm => vm.setData(err, post))
})
},
// when route changes and this component is already rendered,
// the logic will be slightly different.
beforeRouteUpdate (to, from, next) {
this.post = null
getPost(to.params.id, (err, post) => {
this.setData(err, post)
next()
})
},
methods: {
setData (err, post) {
if (err) {
this.error = err.toString()
} else {
this.post = post
}
}
}
}
9.1.13.14. Lazy loading
Only load components when they are triggered by navigation
const Foo = () => Promise.resolve({ /* component definition */ })
import('./Foo.vue') // returns a Promise
Webpack
const Foo = () => import('./Foo.vue')
// everything is the same in router config
const router = new VueRouter({
routes: [
{ path: '/foo', component: Foo }
]
})
Group multiple components and lazy load them
const Foo = () => import(/* webpackChunkName: "group-foo" */ './Foo.vue') const Bar = () => import(/* webpackChunkName: "group-foo" */ './Bar.vue') const Baz = () => import(/* webpackChunkName: "group-foo" */ './Baz.vue')
9.1.14. State Management, Vuex
9.1.14.1. Basics, option:state
vuex is a vue:plugin
npm install vuex –save
./src/store/store.js (or index.js)
import Vue from 'vue'
import Vuex from 'vuex'
Vue.use(Vuex)
export default new Vuex.Store({
state: {
count: 0
},
mutations: {
increment: state => state.count++, // shorthand for
// increment (state) { state.count++ }
decrement: state => state.count--
},
actions: {
// actions can be defined directly here to so that
// they can interact with other actions/modules
}
})
./src/main.js
import Vue from 'vue'
import App from './App'
import router from './router'
import store from './store/store' // new, if store/index.js is used, use ./store
Vue.config.productionTip = false
/* eslint-disable no-new */
new Vue({
el: '#app',
router,
store, // new
template: '<App/>',
components: { App }
})
App.vue
<template>
<div id="app">
<img src="./assets/logo.png">
<router-view/>
<p>{{ count }}</p>
<p>
<button @click="increment">+</button>
<button @click="decrement">-</button>
</p>
</div>
</template>
<script>
export default {
name: 'app',
computed: {
count () {
return this.$store.state.count
}
},
methods: {
increment () {
this.$store.commit('increment')
},
decrement () {
this.$store.commit('decrement')
}
}
}
</script>
9.1.14.2. mapState in component computed props
mapState is to get state as it is without calculation or maybe has to do calculations with local state. It can save some keystrokes: Before App.vue
<script>
export default {
name: 'app',
computed: {
count () {
return this.$store.state.count
}
},
// some methods: {}
}
</script>
Use mapState
// in full builds helpers are exposed as Vuex.mapState
import { mapState } from 'vuex'
export default {
// ...
computed: mapState({
// arrow functions can make the code very succinct!
count: state => state.count,
// passing the string value 'count' is same as `state => state.count`
countAlias: 'count',
// to access local state with `this`, a normal function must be used
countPlusLocalState (state) {
return state.count + this.localCount
}
})
}
Use array in mapState to map this.count to store.state.count
computed: mapState([ // map this.count to store.state.count 'count' ])
mapState returns an object inside the component's computed. How to combine with other computed properties?
computed: {
localComputed () { /* ... */ },
// mix this into the outer object with the object spread operator
...mapState({
// ...
})
}
9.1.14.3. option:getters, mapGetters in component computed props
getters or mapGetters are to get state and compute on it.
./src/store/store.js
const store = new Vuex.Store({
state: {
todos: [
{ id: 1, text: '...', done: true },
{ id: 2, text: '...', done: false }
]
},
getters: {
doneTodos: state => {
return state.todos.filter(todo => todo.done)
},
doneTodosCount: (state, getters) => {
return getters.doneTodos.length
},
// pass parameter
getTodoById: (state, getters) => (id) => {
return state.todos.find(todo => todo.id === id)
}
}
})
store.getters.doneTodos // -> [{ id: 1, text: '...', done: true }]
store.getters.doneTodosCount // -> 1
store.getters.getTodoById(2) // -> { id: 2, text: '...', done: false }
In component
computed: {
doneTodosCount () {
return this.$store.getters.doneTodosCount
}
}
mapGetters in component computed props
import { mapGetters } from 'vuex'
export default {
// ...
computed: {
// mix the getters into computed with object spread operator
...mapGetters([
'doneTodosCount',
'anotherGetter',
// ...
])
}
}
Change name
...mapGetters({
// map `this.doneCount` to `store.getters.doneTodosCount`
doneCount: 'doneTodosCount'
})
9.1.14.4. option:mutations, commit/mapMutations in component, Synchronous vue:plugin:vuex:mutations
Mutation is the only way to actually change state
Mutation is defined in Vuex and it has to be synchronous!
Add new prop to state
Vue.set(obj, 'newProp', 123)
// or rebuild the whole state
state.obj = { ...state.obj, newProp: 123 }
- Mutation in store, commit in component
export default new Vuex.Store({ state: { count: 1 }, mutations: { increment: state => state.count++, // shorthand for // increment (state) { state.count++ } } })commit in component
methods: { increment () { this.$store.commit('increment') }, decrement () { this.$store.commit('decrement') } }Mutation-Commit with payload
mutations: { increment (state, payload) { state.count += payload.amount } } // in component methods: { increment: () { this.$store.commit('increment', { amount: 10 }) // object commit form /* this.$store.commit({ type: 'increment', amound: 10 }) */ } } - mapMutations in component
save typing in component without typing commit
import { mapMutations } from 'vuex' export default { // ... methods: { ...mapMutations([ 'increment', // map `this.increment()` to `this.$store.commit('increment')` // `mapMutations` also supports payloads: 'incrementBy' // map `this.incrementBy(amount)` to `this.$store.commit('incrementBy', amount)` ]), ...mapMutations({ add: 'increment' // map `this.add()` to `this.$store.commit('increment')` }) } } - Constant for Mutation
// mutation-types.js export const SOME_MUTATION = 'SOME_MUTATION' // store.js import Vuex from 'vuex' import { SOME_MUTATION } from './mutation-types' const store = new Vuex.Store({ state: { ... }, mutations: { // we can use the ES2015 computed property name feature // to use a constant as the function name [SOME_MUTATION] (state) { // mutate state } } })
9.1.14.5. option:actions with context, dispatch/mapActions in component, Asynchronous
Actions commit mutations
- Basics
const store = new Vuex.Store({ state: { count: 0 }, mutations: { increment (state) { state.count++ } }, actions: { increment (context) { // context exposes the same set of methods/props on the store instance // e.g. context.state, context.getters // but context is not the store instance context.commit('increment') } /* the above can be shortened to increment ({ commit }) { commit('increment') } */ } })In component
this.$store.dispatch('increment') - mapActions and Asynchronous
store.js
actions: { // checkout only needs commit and state from context // products is a payload checkout ({ commit, state }, products) { // save the items currently in the cart const savedCartItems = [...state.cart.added] // send out checkout request, and optimistically // clear the cart commit(types.CHECKOUT_REQUEST) // the shop API accepts a success callback and a failure callback shop.buyProducts( products, // handle success () => commit(types.CHECKOUT_SUCCESS), // handle failure () => commit(types.CHECKOUT_FAILURE, savedCartItems) ) } }In component
import { mapActions } from 'vuex' export default { // ... methods: { ...mapActions([ 'increment', // map `this.increment()` to `this.$store.dispatch('increment')` // `mapActions` also supports payloads: 'incrementBy' // map `this.incrementBy(amount)` to `this.$store.dispatch('incrementBy', amount)` ]), ...mapActions({ add: 'increment' // map `this.add()` to `this.$store.dispatch('increment')` }) } } - Series of actions, return Promise
actions: { actionA ({ commit }) { return new Promise((resolve, reject) => { setTimeout(() => { commit('someMutation') resolve() }, 1000) }) }, actionB ({ dispatch, commit }) { return dispatch('actionA').then(() => { commit('someOtherMutation') }) } }store.dispatch('actionA').then(() => { // ... }) - Series of actions, async/await
- See async function
// assuming `getData()` and `getOtherData()` return Promises
actions: { async actionA ({ commit }) { commit('gotData', await getData()) }, async actionB ({ dispatch, commit }) { await dispatch('actionA') // wait for `actionA` to finish commit('gotOtherData', await getOtherData()) } }
9.1.14.6. option:modules
const moduleA = {
state: { ... },
mutations: { ... },
actions: { ... },
getters: { ... }
}
const moduleB = {
state: { ... },
mutations: { ... },
actions: { ... }
}
const store = new Vuex.Store({
modules: {
a: moduleA,
b: moduleB
}
})
store.state.a // -> `moduleA`'s state
store.state.b // -> `moduleB`'s state
Register a module after the store has been created
// register a module `myModule`
store.registerModule('myModule', {
// ...
})
// register a nested module `nested/myModule`
store.registerModule(['nested', 'myModule'], {
// ...
})
store.unregisterModule(moduleName)
- Module local state and rootState
const moduleA = { state: { count: 0 }, mutations: { increment (state) { // `state` is the local module state state.count++ } }, getters: { doubleCount (state, rootState) { // rootState is root state return state.count * 2 + rootState.count } } actions: { incrementIfOddOnRootSum ({ state, commit, rootState }) { // local state if ((state.count + rootState.count) % 2 === 1) { commit('increment') } } } } - Namespace
Namespaced getters and actions will receive localized getters, dispatch and commit. In other words, you can use the module assets without writing prefix in the same module.
const store = new Vuex.Store({ modules: { account: { namespaced: true, // module assets state: { ... }, // module state is already nested and not affected by namespace option getters: { isAdmin () { ... } // -> getters['account/isAdmin'] }, actions: { login () { ... } // -> dispatch('account/login') }, mutations: { login () { ... } // -> commit('account/login') }, // nested modules modules: { // inherits the namespace from parent module myPage: { state: { ... }, getters: { profile () { ... } // -> getters['account/profile'] } }, // further nest the namespace posts: { namespaced: true, state: { ... }, getters: { popular () { ... } // -> getters['account/posts/popular'] } } } } } })In component
computed: { ...mapState({ a: state => state.some.nested.module.a, b: state => state.some.nested.module.b }) }, methods: { ...mapActions([ 'some/nested/module/foo', 'some/nested/module/bar' ]) }Or this
computed: { ...mapState('some/nested/module', { a: state => state.a, b: state => state.b }) }, methods: { ...mapActions('some/nested/module', [ 'foo', 'bar' ]) }Or
import { createNamespacedHelpers } from 'vuex' const { mapState, mapActions } = createNamespacedHelpers('some/nested/module') export default { computed: { // look up in `some/nested/module` ...mapState({ a: state => state.a, b: state => state.b }) }, methods: { // look up in `some/nested/module` ...mapActions([ 'foo', 'bar' ]) } } - rootGetters
modules: { foo: { namespaced: true, getters: { // `getters` is localized to this module's getters // you can use rootGetters via 4th argument of getters someGetter (state, getters, rootState, rootGetters) { getters.someOtherGetter // -> 'foo/someOtherGetter' rootGetters.someOtherGetter // -> 'someOtherGetter' }, someOtherGetter: state => { ... } }, actions: { // dispatch and commit are also localized for this module // they will accept `root` option for the root dispatch/commit someAction ({ dispatch, commit, getters, rootGetters }) { getters.someGetter // -> 'foo/someGetter' rootGetters.someGetter // -> 'someGetter' dispatch('someOtherAction') // -> 'foo/someOtherAction' dispatch('someOtherAction', null, { root: true }) // -> 'someOtherAction' commit('someMutation') // -> 'foo/someMutation' commit('someMutation', null, { root: true }) // -> 'someMutation' }, someOtherAction (ctx, payload) { ... } } } } - Namespace in Plugin Development vue:plugin:vuex:namespace in plugin
// get namespace value via plugin option // and returns Vuex plugin function export function createPlugin (options = {}) { return function (store) { // add namespace to plugin module's types const namespace = options.namespace || '' store.dispatch(namespace + 'pluginAction') } }
9.1.14.7. option:strict, strict mode
In strict mode, whenever Vuex state is mutated outside of mutation handlers, an error will be thrown. This ensures that all state mutations can be explicitly tracked by debugging tools.
const store = new Vuex.Store({
// ...
strict: process.env.NODE_ENV !== 'production'
})
9.1.14.8. option:watch
watch(getter: Function, cb: Function, options?: Object)
Reactively watch a getter function's return value, and call the callback when the value changes. The getter receives the store's state as the only argument. Accepts an optional options object that takes the same options as Vue's vm.$watch method.
To stop watching, call the returned handle function.
9.1.14.9. option:plugin
plugins option that exposes hooks for each mutation. A Vuex plugin is simply a function that receives the store as the only argument
const myPlugin = store => {
// called when the store is initialized
store.subscribe((mutation, state) => {
// called after every mutation.
// The mutation comes in the format of `{ type, payload }`.
})
}
const store = new Vuex.Store({
// ...
plugins: [myPlugin]
})
9.1.14.10. App Structure, Example
├── index.html
├── main.js
├── api
│ └── ... # abstractions for making API requests
├── components
│ ├── App.vue
│ └── ...
└── store
├── index.js # where we assemble modules and export the store
├── actions.js # root actions
├── mutations.js # root mutations
└── modules
├── cart.js # cart module
└── products.js # products module
./api/shop.js
/**
* Mocking client-server processing
*/
const _products = [
{"id": 1, "title": "iPad 4 Mini", "price": 500.01, "inventory": 2},
{"id": 2, "title": "H&M T-Shirt White", "price": 10.99, "inventory": 10},
{"id": 3, "title": "Charli XCX - Sucker CD", "price": 19.99, "inventory": 5}
]
export default {
getProducts (cb) {
setTimeout(() => cb(_products), 100)
},
buyProducts (products, cb, errorCb) {
setTimeout(() => {
// simulate random checkout failure.
(Math.random() > 0.5 || navigator.userAgent.indexOf('PhantomJS') > -1)
? cb()
: errorCb()
}, 100)
}
}
./store/index.js
import Vue from 'vue' import Vuex from 'vuex' import * as actions from './actions' import * as getters from './getters' import cart from './modules/cart' import products from './modules/products' import createLogger from '../../../src/plugins/logger' Vue.use(Vuex) const debug = process.env.NODE_ENV !== 'production' export default new Vuex.Store({ actions, getters, modules: { cart, products }, strict: debug, plugins: debug ? [createLogger()] : [] })
./store/modules/products.js
import shop from '../../api/shop' import * as types from '../mutation-types' // initial state const state = { all: [] } // getters const getters = { allProducts: state => state.all } // actions const actions = { getAllProducts ({ commit }) { shop.getProducts(products => { commit(types.RECEIVE_PRODUCTS, { products }) }) } } // mutations const mutations = { [types.RECEIVE_PRODUCTS] (state, { products }) { state.all = products }, [types.ADD_TO_CART] (state, { id }) { state.all.find(p => p.id === id).inventory-- } } export default { state, getters, actions, mutations }
./components/App.vue
<template>
<div id="app">
<h1>Shopping Cart Example</h1>
<hr>
<h2>Products</h2>
<product-list></product-list>
<hr>
<cart></cart>
</div>
</template>
<script>
import ProductList from './ProductList.vue'
import Cart from './Cart.vue'
export default {
components: { ProductList, Cart }
}
</script>
./component/ProductList.vue
<template>
<ul>
<li v-for="p in products">
{{ p.title }} - {{ p.price | currency }}
<br>
<button
:disabled="!p.inventory"
@click="addToCart(p)">
Add to cart
</button>
</li>
</ul>
</template>
<script>
import { mapGetters, mapActions } from 'vuex'
export default {
computed: mapGetters({
products: 'allProducts'
}),
methods: mapActions([
'addToCart'
]),
created () {
this.$store.dispatch('getAllProducts')
}
}
</script>
9.1.15. vue-cli vue:vue-cli
9.1.15.1. Basics
# If vue-cli is previously installed, uninstall it first # npm install -g vue-cli # npm uninstall vue-cli -g # or # yarn global remove vue-cli # @vue/cli requires NPM 8.11 npm install -g @vue/cli # yarn global add @vue/cli # check version. Should be 3.x vue --version # create a project in a new dir vue create hello-world # create a project inside a dir vue create . # Default preset: babel and ESLint # Manual select features for preset. Then a JSON file `~/.vuerc` is created to store the preset vue create --help vue ui # install a Vue CLI plugin. This resolves @vue/eslint to the full pkg name @vue/cli-plugin-eslint, install it from npm and invokes its generator vue adde eslint # same as above vue add cli-plugin-eslint
9.1.15.2. Services
@vue/cli-serviceis installedvue-cli-service- (no term)
scriptsinpackage.json- serve
vue-cli-service serve
9.1.15.3. Typescript
- Manual select preset
- Babel, TypeScript, CSS Pre-processors, Linter / Formatter
- Use class-style component syntax
Yes Classic Vue Syntax vs Class-Style Component Syntax
datais a function vsdata.messageis a property Computed properties become getter #+NAME Classic Vue Syntaximport Vue, { VNode } from 'vue' export const HelloComponent = Vue.extend({ data () { return { message: 'Hello', } }, methods: { greet (): string { return this.message + ' world'; } }, computed: { greeting(): string { return this.greet() + '!'; } }, render (createElement): VNode { return createElement('div', this.greeting); } });
#+NAME Class-Style Component Syntax
import Vue from 'vue' import Component from 'vue-class-component' @Component({ template: '<div></div>' }) export default class HelloComponent extends Vue { message: string = 'Hello' greet(): string { return this.message + ' world'; } get greeting() { return this.greet() + '!'; } }
- (no term)
- Use Babel alongside TypeScript for auto-detected polyfills? Yes (default)
- Pick a CSS pre-processer (PostCSS, Autoprefixer and CSS Modules are supported by default)
- Sass/SCSS (with node-sass)
- Pick a linter / formatter config
- TSLint
- Pick additional lint features
- Lint on save
- Where do you prefer placing config for Babel, PostCSS, ESLint, etc.?
- In dedicated config files. Want to separate them instead of cluttering
package.json- tslint.json
- tsconfig.json
- postcss.config.js
- babel.config.js
- (no term)
- Directories
- public
- src
- main.ts
- runs first
9.1.16. Dev - Webpack Template
9.1.16.1. Basics
Install node module vue:vue-cli
vue init webpack my-project cd my-project npm install npm run devOther template can be used
# vue init <template-name> <project-name> vue init webpack-simple my-project # Template on github vue init username/repo my-project # Template on Bitbucket vue init bitbucket:username/repo my-project # for a tag or branch vue init bitbucket:username/repo#branch my-project # local template vue init ~/fs/path/to-custom-template my-project
9.1.16.2. npm
Default port is 8080 and it can be auto adjusted. It loads the vuejs-templates/webpack npm run dev includes
- webpack + vue-loader for single file Vue components
- state preserving hot-reload
- state preserving compilation error overlay
- Lint-on-save with ESLint
- Source maps
npm run build :: build for production
- UglifyJS
- html-minifier https://github.com/kangax/html-minifier
- CSS across all components extracted into a single file and minified with cssnano
- Static assets compiled with versio hashes for efficient long-term caching and an auto-generated production index.html with proper URLs to these generated assets
- Use
npm run build --reportto build with bundle size analytics
npm run unit :: unit tests run in PhantomJS with Karma + Mocha + karma-webpack
- support ES2015+ in test files
- support all webpack loaders
- easy mock injection
num run e2e :: end-to-end tests with Nightwatch
- run tests in multiple browsers in parallel
- works with one command out of the box
- Selenium and chromedrive dependencies automatically handled
- automatically spawns the Selenium Server
9.1.16.3. Structure
. ├── build/ # webpack config files for dev and prod. Don't touch unless to customize Webpack loaders │ └── ... # e.g. webpack.base.conf.js ├── config/ │ ├── index.js # main project config. vue:webpack:integrate backend │ └── ... ├── src/ # app files │ ├── main.js # app entry file │ ├── App.vue # main app component │ ├── components/ # ui components │ │ └── ... │ └── assets/ # module assets (processed by webpack) │ └── ... ├── static/ # pure static assets (directly copied and not processed by Webpack) ├── test/ │ └── unit/ # unit tests │ │ ├── specs/ # test spec files │ │ ├── index.js # test build entry file │ │ └── karma.conf.js # test runner config file │ └── e2e/ # e2e tests │ │ ├── specs/ # test spec files │ │ ├── custom-assertions/ # custom assertions for e2e tests │ │ ├── runner.js # test runner script │ │ └── nightwatch.conf.js # test runner config file ├── .babelrc # babel config ├── .postcssrc.js # postcss config ├── .eslintrc.js # eslint config ├── .editorconfig # editor config ├── index.html # index.html template. Webpack will generate assets and auto inject into it during dev and builds. └── package.json # build scripts and dependencies
9.1.16.4. Linter configuration
Modify ./.eslintrc.js Add under rule
"semi": [2, "always"]
Preset standard is set. Checkout all the rules.
9.1.16.5. CSS Pre-processor
This boilerplate has pre-configured LESS, SASS, Stylus adn PostCSS. Just install the appropriate webpack loader.
npm install sass-loader node-sass --save-dev
After, in *.vue components
<style lang="scss"> /* write SASS! */ </style>
lang="scss" corresponds to the CSS-superset syntax (with curly braces and semicolons). lang="sass" corresponds to the indentation-based syntax.
Standalone CSS Files Import global and custom style files from your root App.vue component
<!-- App.vue --> <style src="./styles/global.less" lang="less"></style>
For other 3rd party, e.g. Bootstrap, place them inside /static and reference them directly in index.html.
9.1.16.6. Processed Static Assets => /src/assets
In *.vue components, all templates and CSS are parsed by vue-html-loader and css-loader to look for asset URLs.
<img src="./logo.png"> and background: url(./logo.png), is a relative asset path and will be resolved by Webpack as a module dependency.
These assets may be inlined/copied/renamed during build.
Recommended :: put each component in its own directory with its static assets right next to it and continue to use relative path.
- Relative URLs
- e.g. ./assets/logo.png will be interpreted as a module dependency. They will be replaced with an auto-generated URL based on your Webpack output configuration.
- Non-prefixed URLs
- e.g. assets/logo.png will be treated the same as the relative URLs and translated into ./assets/logo.png.
- URLs prefixed with ~
- are treated as a module request, similar to require('some-module/image.png'). You need to use this prefix if you want to leverage Webpack's module resolving configurations. For example if you have a resolve alias for assets, you need to use <img src="~assets/logo.png"> to ensure that alias is respected.
- Root-relative URLs
- e.g. /assets/logo.png are not processed at all.
Files in ./src/assets/ or any webpack processed assets may be copied to \((build.assetsRoot)\)(build.assetsSubDirectory)
If assets are inserted via JavaScript like computed property, you need to do it in this way
computed: {
background () {
return require('./bgs/' + this.id + '.jpg')
}
}
Limitation :: all images under ./bgs/ will be included in the final build.
Files in ./static/ will be directly copied to \((build.assetsPublicPath)\)(build.assetsSubDirectory). Refer to vue:webpack:config:index.js
Files in ./static/ should be referenced as absolute path like static[filename]. If $(build.assetsSubDirectory) is changed, you need to change in the reference as well.
9.1.16.7. Integrate with Backend vue:webpack:integrate backend
- Proxy requests to a backend
In
config/index.js, edit dev.proxyTable option Proxy the request /api/posts/1 to http://jsonplaceholder.typicode.com/posts/1.module.exports = { // ... dev: { proxyTable: { // proxy all requests starting with /api to jsonplaceholder '/api': { target: 'http://jsonplaceholder.typicode.com', changeOrigin: true, pathRewrite: { '^/api': '' } } } } }It uses http-proxy-middleware
proxyTable: { '**': { target: 'http://jsonplaceholder.typicode.com', filter: function (pathname, req) { return pathname.match('^/api') && req.method === 'GET' } } } - Backend path
Default ./config/index.js vue:webpack:config:index.js
var path = require('path') module.exports = { build: { index: path.resolve(__dirname, 'dist/index.html'), // has to be absolute // e.g. path.resolve(__dirname, 'resources/views/index.blade.php'), assetsRoot: path.resolve(__dirname, 'dist'), // final after build is /dist/static // path.resolve(__dirname, 'public/'); assetsSubDirectory: 'static', // changing this also means absolute paths for real static assets need to be changed assetsPublicPath: '/', productionSourceMap: true }, dev: { port: 8080, proxyTable: {} } }
9.1.16.8. .vue file vue-loader
.vue file name
Add attribute lang to chnage language :: <style lang="sass">
<template> :: default html, 0 or 1 template block. <script> :: default js (ES2015 if babel-loader or buble-loader exists). - or 1 script block. require() can be used. With ES2015, import and export can also be used. The result is to export a Vue.js component option object. An extended constructor created by Vue.extend() can also be exported.
<style> :: default css.
- Multiple blocks.
- Can have scoped or mobile attributes.
- vue:loader:scoped css, vue:loader:css modules
- can have mixed encapsulation modes
- scoped css vue:loader:scoped css
Scoped to current component.
<style scoped> .example { color: red; } </style> <template> <div class="example">hi</div> </template>Parent component's styles will not leak into child components. However, a child component's root node will be affected by both the parent's scoped CSS and the child's scoped CSS. So that the parent can style the child root element for layout purpose.
Deep selector
<style scoped> .a >>> .b { /* ... */ } </style> // will be compiled into .a[data-v-f3f3eg9] .b { /* ... */ } // may use /deep/ combinator instead for >>> in other pre-processors, like SASS - CSS Modules
9.1.17. Ajax, axios
https://github.com/axios/axios https://rubyplus.com/articles/5051-Using-vue-cli-and-axios-in-Vue-js-2
npm install axios --save
// just import it and use it wherever you want
<script>
import axios from 'axios'
export default {
name: 'app',
data: function () {
return {
articles: []
}
},
methods: {
fetchArticles: function () {
axios.get('http://localhost:3000/articles').then((response) => {
this.articles = response.data
}, (error) => {
console.log(error)
})
}
},
mounted: function () {
this.fetchArticles()
}
}
</script>
9.2. jQuery
9.2.1. Version, Closure, document ready
jQuery.fn.jquery
- Closure
(function($){ // can do something like $.fn.function_name = function(x){}; })(jQuery);
- Document ready shorthand
$(function(){
});
<script src="other_lib.js"></script> <script src="jquery.js"></script> <script> $.noConflict() jQuery(document).ready(function ($) { // Code that uses jQuery's $ can follow here. }) // Code that uses other library's $ can follow here. </script> <script> jQuery.noConflict(); (function ($) { $(function () { // More code using $ as alias to jQuery }) })(jQuery) // Other code using $ as an alias to the other library </script>
9.2.1.1. jQuery.noConflict
Return control of $ back to the other library with a call to $.noConflict(). Old references of $ are saved during jQuery initialization; noConflict() simply restores them.
var j = jQuery.noConflict(); // different alias // Do something with jQuery j( "div p" ).hide(); // Do something with another library's $() $( "content" ).style.display = "none";
Completely move jQuery to a new namespace in another object.
var dom = {}; dom.query = jQuery.noConflict( true ); // If for some reason two versions of jQuery are loaded (which is not recommended), calling $.noConflict( true ) from the second version will return the globally scoped jQuery variables to those of the first version. // Do something with the new jQuery dom.query( "div p" ).hide(); // Do something with another library's $() $( "content" ).style.display = "none"; // Do something with another version of jQuery jQuery( "div > p" ).hide();
9.2.2. Ajax
$.get(url[, data] [, success], [, dataType] )$.post(url[, data] [, success], [, dataType] )
$.ajax({
method: 'POST', // Default: GET
// Prior to 1.9.0, use type instead of method
url: "some.php", // Default current page
data: { name: "a", location: "Boston" } // Data send to server. Converted as URL parameters
// if data is not key/value pairs, change processData to false
processData: true,
// If data is not key/value pairs (object), say it's a DOMDocument, change it to false
contentType: 'applicaton/x-www-form-urlencoded;charset=UTF-8', // Default. MIME type
dataType: "json", // Default is not setting anything. MIME type of response
// > v1.5, this option can convert response to the MIME types you want
// "json xml" :: first it will try to convert to json. if failed, convert ot xml
// "script" :: the response is javascript, and it will be run afer it's received.
// cross domain should be automatically handled
// crossDomain: default false for same-domain requests, true for cross-domain requests
context: document.body, // provide context for callbacks. If not set, this referrence
// in callbacks is the Ajax request settings
cache: true, // You can only change cache for HEAD and GET requests.
headers: {}, // Default. Add (or add to overwrite) request header
// Callback options
beforeSend: function(jqXHR, settings) {
// Modify jqXHR object before it is sent.
// e.g. request header
// For receiving binary string (image or any binary data)
// jqXHR.overrideMimeType("text/plain; charset=x-user-defined");
},
error: function(jqXHR, textStatus, errorThrown) {
// if request fails.
},
dataFilter: function( data, type) {
// raw response data
// dataType option
// You should sanitize the data here so to match the dataType option
},
success: function( data, textStatus, jqXHR ) {
},
complete: function( jqXHR, textStatus) {}
})
/* Promise callbacks: .done, .fail, .always. In those callbacks,
`this` reference is the `context` in the request. If `context` was not set, `this`
will be the Ajax setting.
You can add multiple promise callbacks.
*/
.done(function(data, textStatus, jqXHRmsg) {
// .success() will be deprecated in v3.0
})
.fail(function(jqXHR, textStatus, errorThrown) {
// .error() will be deprecated in v3.0
})
.always(function(data|jqXHR, textStatus, jqXHR|errorThrown) {
// .complete() will be dprecated in v3.0
});
9.2.3. jQuery Selectors, Loop through found elements
You can use all CSS Selectors
http://api.jquery.com/category/selectors/
$('li').each(function(i) { // i = 0, 1, etc. $(this).addClass("foo"); }); // multiple attributes var langEn = $('head link[rel=alternate][hreflang=en]').attr('href');
First matched element :: var iframeId = $("#div-iframe").find('iframe').get(0).id;
9.2.4. iFrame refer to parent
var parentVideoDiv = parent.jQuery('#videoplayer').parent('.the-content');
9.2.5. Find elements
var $querySelection = '...'; if ($querySelect.length > 0) { // not empty // if ($querySelect.length) {} }
9.2.5.1. Get parent element, sibling, closet
var parentDiv = jQuery('#videoplayer').parent('.content'); var siblingDiv = parentDiv.prev(); // preceding sibling var siblingDiv = parentDiv.next(); // next sibling
.closest starts from the current element, travels up the DOM tree until it finds a match for the supplied selector, returns zero or one element.
$( "li.item-a" ) .closest( "ul" ) .css( "background-color", "red" );
9.2.5.2. Find inside an element, .find, .first, .last
siblingDiv.find(".panel.panel-default");
// by data attribute
var $all = $('#id').find('[data-name="' + js_var + '"]');
$first = $all.first();
9.2.6. Pass $(this) to javascript function
$(this)is jQuery object, but this is a normal this
function doFunc(obj) { obj.className = 'hello'; // className is a normal javascript function } $('li').each(function() { doFunc(this); }); // or function handler(event) { var target = $(event.target); if (target.is("li")) { target.children().toggle(); } } $("ul").click(handler).find("ul").hide();
9.2.7. General click to element
Any elements with attribute data-target="goto1" with id #goto1
LiliJs.clickToGo = function(event) { var $target = $(event.target); var iter = 0; var itermax = 10; var goto = ''; while (iter < itermax ) { if ($target.data('target')) { goto =$target.data('target'); break; } else { $target = $target.parent(); iter++; console.log('+1'); } } if (goto) { $("#"+goto).get(0).scrollIntoView(); } } $(function() { $('.click-1, .click-2').click(LiliJs.clickToGo); });
9.2.8. Pointer Events
https://mobiforge.com/design-development/html5-pointer-events-api-combining-touch-mouse-and-pen
| mousedown | touchstart | pointerdown |
| mouseenter | pointerenter | |
| mouseleave | pointerleave | |
| mousemove | touchmove | pointermove |
| mouseout | pointerout | |
| mouseover | pointerover | |
| mouseup | touchend | pointerup |
pointerover :: pointer moves over an element (enters its hit test boundaries) pointerenter :: pointer moves over an element or one of its descendants. Differs to pointerover in that it doesn’t bubble
pointerdown :: active buttons state is entered: for touch and stylus, this is when contact is made with screen; for mouse, when a button is pressed pointermove :: pointer changes coordinates, or when pressure, tilt, or button changes fire no other event pointerup :: active buttons state is left: i.e. stylus or finger leaves the screen, or mouse button released
pointercancel :: pointer is determined to have ended, e.g. in case of orientation change, accidental input e.g. palm rejection, or too many pointers
pointerout :: pointer moves out of an element (leaves its hit test boundaries). Also fired after pointerup event for no-hover supported devices, and after pointercancel event pointerleave :: pointer moves out of an element and its descendants
gotpointercapture :: when an element becomes target of pointer lostpointercapture :: when element loses pointer capture
$('li').on( events [, selector ] [, data ], handler )
There's no .on('hover'). Hover is equal to .on('mouseenter mouseleave')
CSS :hover vs :focus Hover is 'true' when the mouse pointer is over an element. Focus is true if the cursor is in that element. It's possible for hover to be false and focus true (e.g click in a text field then move the mouse away)
9.2.9. Replace class
parentDiv.removeClass('col-md-9').addClass('col-md-12');
Or remove a class if it exists or add a class if it doesn't
$('#id').toggleClass('col-md-9 col-md-12');
Remove class with wild card
$("#hello").removeClass (function (index, className) {
return (className.match(/(^|\s)color-\S+/g) || []).join(' ');
});
9.2.10. Move element
/* Before .siblingDiv .parentDiv */ siblingDiv.insertAfter(parentDiv); /* After .parentDiv .siblingDiv */
9.2.11. Add Style
var selector = jQuery('#id');
selector.attr('style', function(i,s) {
var a = ' visible: visible !important;
if (typeof s !== 'undefined') {
return s + a;
}
else {
return a;
}
});
9.2.12. Form
9.2.12.1. serialize
Checkboxes and radio buttons are included only if they're checked. All seriailized form field elements must have name attribute.
$("#form_id").on("submit", function(e) {
event.preventDefault();
console.log($(this).serialize());
});
9.2.12.2. Disabled Field
Disabled fields are not editable, selectable, copyable and they are not submitted. In order to select and copy value of a disabled field, you need to enable it when mouseover and disable it when mouseout.
Can't register events directly on disabled fields. Events can be registered on its parent
$('.parent_container')
.mouseover(function() { $(this).find('textarea').prop('disabled',false)})
.mouseout(function() { $(this).find('textarea').prop('disabled',true)});
9.2.13. Animate
9.2.13.1. Scroll
var offset = $("[ng-controller=aController]").offset();
offset.top -= 100; // offset.left is also available
$("html,body").animate({
scrollTop: offset.top,
// scrollLeft: offset.left // if it doesn't change, it won't have to be set
});
9.2.14. Viewport Change Event
js:viewport size \((window).resize(function() { consoel.log(\)(window).width()); // integer without unit });
9.2.15. jQuery UI: Dialog jqueryui:dialog
<div id="interstitial">
<!-- script tag inside the container will be run every time the dialog is opend!
Consider remove script tag before .dialog and reattach javascript later.
Content inside the container will be removed and then added to div.ui-dialog at the bottom of DOM.
-->
</div>
jQuery(function(){
jQuery("#interstitial").dialog({
width: 'auto', /* no scrollbar, auto fit content */
height:'auto',
resizable: false,
draggable: false, /* movable */
closeOnEscape: true,
/* Add extra buttons below the content
buttons: [{
text: 'ok',
icons: {primary: "ui-icon-closethick"},
click: function() {jQuery(this).dialog("close");}
}],
*/
modal: true, /* add background to isolate the modal box */
create: function (event, ui) {
jQuery("#ui-dialog-title-dialog").hide();
/* custom background/overlay */
jQuery(".ui-widget-content").css({"background-color":"transparent","background":"transparent","border":"none"});
/* title bar is transparent */
jQuery(".ui-dialog-titlebar").css({"background-color":"transparent","background":"transparent","border":"none"});
jQuery(".ui-dialog-titlebar").removeClass('ui-widget-header');
// Change the default title & close button to a custom link
//jQuery(".ui-dialog-titlebar").html('<a href="#" role="button"><span class="myCloseIcon" style="color:white;">X close</span></a>')
// You can still load/modify the content here and the width/height will still be auto fit
},
open: function() {
/* custom background */
jQuery('.ui-widget-overlay').addClass('custom-overlay');
/* Customize the default close button: only X displays */
jQuery(this).closest(".ui-dialog")
.find(".ui-dialog-titlebar-close")
.removeClass("ui-dialog-titlebar-close")
.html("<span class='ui-button-icon-primary ui-icon ui-icon-closethick'></span>");
jQuery(".ui-dialog-titlebar button").css({"border":"none"});
/* Close dialog when click outside */
jQuery('.ui-widget-overlay').bind('click',function(){
jQuery("#interstitial").dialog('close');
})
},
close: function() {
/* custom background/overlay */
/*
* .ui-widget-overlay.custom-overlay
{
background-color: black;
background-image: none;
opacity: 0.9;
}
* */
jQuery('.ui-widget-overlay').removeClass('custom-overlay');
}
});
/* bind events for custom buttons created
jQuery("span.myCloseIcon").click(function() {
jQuery("#interstitial").dialog( "close" );
});
*/
});
9.2.16. jQuery UI: Sortable
$(function() {
$("#slideshow-pictures").sortable({
items:'.dz-preview',
cursor: 'move',
opacity: 0.5,
containment: '#slideshow-pictures',
distance: 20,
tolerance: 'pointer'
});
});
9.3. TweenLite, TweenMax, Greensock
TweenMax has other common plugins included e.g. CSSPlugin where you can ease CSS property changes.
- CSSPlugin
- change individual DOM element's style attribute
- (no term)
- RoundPropsPlugin
- (no term)
- BezierPlugin
- (no term)
- AttrPlugin
- (no term)
- DirectionalRotationPlugin
- (no term)
- EasePack
- (no term)
- TimelineLite
- (no term)
- TimelineMax
Extra plugins which are separate js files loaded after TweenMax https://greensock.com/tweenmax:
- CSSRulePlugin
- change style sheet rules e.g. for class .myClass and other pseudo elements :after :before which are impossible to reference directly in JavaScript
- (no term)
- modifiers
- (no term)
- text
- (no term)
- scrollTo
- (no term)
- pixi
- (no term)
- easel
- Draggable
- Spin, Scrll
- (no term)
- jquery.gsap.js
Using TweenLite only and you'll have to manually include these plugins.
9.3.1. TweenLite.to() TweenLite.from() TweenLite.fromTo()
https://greensock.com/docs/TweenLite/static.to
TweenLite.to( [target object], [duration in seconds], [destination values] );
TweenLite.to(active_slide, 0.5, {marginLeft:'-50%',ease:Power3.easeOut});
TweenLite.to("#myID", 1, {left:"100px"});
// TweenLite will use selector engine e.g. jQuery if present or document.querySelectorAll() or lastly document.getElementById()
TweenLite.to(".myClass", 2, {boxShadow:"0px 0px 20px red", color:"#FC0"});
// multiple objects
TweenLite.to([obj1, obj2, obj3], 1, {opacity:0.5, rotation:45});
TweenLite.from() define the starting values instead of the ending values. TweenLite.fromTo() :: TweenLite.fromTo( target:Object, duration:Number, fromVars:Object, toVars:Object )
var tween = new TweenLite(element, 2, {width:200, height:150});
// or
var tween = TweenLite.to(element, 2, {width:200, height:150});
9.3.1.1. dest:css, relative value
It's important that you remove all dashes (-) from the css property names and use camelCase.
TweenLite.to(logo, 2, {left:"542px",
backgroundColor:"black",
borderBottomColor:"#90e500",
color:"white"});
Adding a += or -= prefix tells GSAP to treat a value as relative to whatever it is when the tween renders the first time. For example, if "left" is currently 50px and you tween to "+=100px", it will end up at 150px. Don't forget to put quotes around the value.
TweenLite.to(logo, 0.5, {left:"+=100px"});
9.3.1.2. dest:ease, easeIn (begin), easeOut (the end), easeInOut (both)
https://greensock.com/docs/Easing
Default is Quad.easeOut
- Linear
- 0 is linear
- Quad
- Cubic
- Quart
- Quint
- Strong
Extras
- Elastic Back Bounce SlowMo SteppedEase Rough Circ Expo Sine
Variants
- .easeIn .easeOut .easeInOut .easeNone
For linear animation, use Linear.easeNone
Sample
TweenLite.to(logo, 3, {left:"440px", ease:Bounce.easeOut});
ease:Bounce.easeOut
ease:"Strong.easeOut"
ease:"easeOutStrong"
9.3.1.3. dest:delay in seconds, dest:paused bool, dest:immediateRender
- paused
- If true, the animation will pause immediately upon creation
- immediateRender
9.3.1.4. dest:onComplete onCompleteParams
TweenLite.to(element, 1, {left:100, onComplete:myFunction});
TweenLite.to(element, 1, {left:100, onComplete:myFunction, onCompleteParams:[element, "param2"]});
// pass the tween instance use {self}
TweenLite.to(element, 1, {left:100, onComplete:myFunction, onCompleteParams:["{self}", "param2"]});
9.3.2. Use Case
9.3.2.1. Slideshow
<section id="slider-wrap" class="home-slide inner">
<div id="slides-frame">
<div id="active-slide" class="slide-img"
style="background:transparent url('1.jpg') no-repeat center 15%;background-size:cover;-webkit-transform: translateZ(0);">
<div class="background-screen">
<div class="page-wrap">
<div class="slide-headline">
<h2>Slide #1 Title</h2>
<p class="quote">Slide #1 Paragraph</p>
<p class="quote-name">Slide #1 Paragraph</p>
<a href="#">Slide #1 Button</a>
</div>
</div>
</div>
</div>
<div id="next-slide" class="slide-img"
style="background:transparent url('2.jpg') no-repeat center 25%;background-size:cover;-webkit-transform: translateZ(0);">
<div class="background-screen">
<div class="page-wrap">
<div class="slide-headline">
<h2>Slide #2 Title</h2>
<p class="quote">Slide #2 Paragraph</p>
<p class="quote-name">Slide #2 Paragraph</p>
<a href="#">Slide #2 Button</a>
</div>
</div>
</div>
</div> <!-- #next-slide -->
</div> <!-- #slides-frame -->
</section>
<!-- @@ Slideshow Data: -->
<div id="slideshow-data" style="display:none!important;">
<div class="slideshow-data-item"
style="background:transparent url('1.jpg') no-repeat center 15%;background-size:cover;-webkit-transform: translateZ(0);">
<div class="background-screen">
<div class="page-wrap">
<div class="slide-headline">
<h2>Slide #1 Title</h2>
<p class="quote">Slide #1 Paragraph</p>
<p class="quote-name">Slide #1 Paragraph</p>
<a href="#">Slide #1 Button</a>
</div>
</div>
</div>
</div>
<div class="slideshow-data-item"
style="background:transparent url('2.jpg') no-repeat center 25%;background-size:cover;-webkit-transform: translateZ(0);">
<div class="background-screen">
<div class="page-wrap">
<div class="slide-headline">
<h2>Slide #2 Title</h2>
<p class="quote">Slide #2 Paragraph</p>
<p class="quote-name">Slide #2 Paragraph</p>
<a href="#">Slide #2 Button</a>
</div>
</div>
</div>
</div>
<div class="slideshow-data-item"
style="background:transparent url('3.jpg') no-repeat center left;background-size:cover;-webkit-transform: translateZ(0);">
<div class="background-screen">
<div class="page-wrap">
<div class="slide-headline">
<h2>Slide #3 Title</h2>
<p class="quote">Slide #3 Paragraph</p>
<p class="quote-name">Slide #3 Paragraph</p>
<a href="#">Slide #3 Button</a>
</div>
</div>
</div>
</div>
</div> <!-- #slideshow-data -->
.page {
max-width:1200px;
width:90%;
margin:0 auto;
}
.home-slide.inner {
height: 550px; /* media query change height */
overflow: hidden;
}
#slides-frame {
width:200%!important;
position:relative;
height:550px;
}
#active-slide, #next-slide {
position:relative;
display:inline-block;
width:50%;
height:100%;
vertical-align:top;
}
#slides-frame .background-screen {
position:relative;
height:100%;
width:100%;
overflow-y:hidden;
background:rgba(0,0,0,0.35); /* darken bg image */
}
.slide-wrap {
/* Text content width */
width:1200px; /* media query change width */
/* width:100%; */
height:100%;
overflow:visible;
margin:0 auto;
}
.slide-headline {
position:absolute;
left:0;
bottom:0;
margin-bottom:155px; /* media query change margin-bottom */
}
window.onload = init();
function init() {
slideshow_checker();
}
function slideshow_checker() {
var slideshow_data = document.getElementById('slider-wrap');
if (slideshow_data != null) {
slideshow(0, 'time', 7000);
}
}
var timer;
function slideshow(id, type, time_length) {
var active_slide = document.getElementById('active-slide'),
active_slide_style = active_slide.getAttribute('style');
var next_slide = document.getElementById('next-slide');
/* -- COUNT SLIDE NUM -- */
var slideshow_data = document.getElementById('slideshow-data');
var slides = slideshow_data.getElementsByClassName('slideshow-data-item');
var slide_count = slides.length;
var nxt_id, nxt_nxt_id, prev_id;
nxt_id = (id + 1);
prev_id = (id - 1);
if (nxt_id == slide_count) {
nxt_id = 0;
}
if ((id == 0)) {
prev_id = (slide_count - 1);
}
// arr_left.setAttribute('onclick','javascript:get_slide('+prev_id+')');
// arr_right.setAttribute('onclick','javascript:get_slide('+nxt_id+')');
if (type == 'click') {
var next_slide_data = slides[id].getAttribute('style');
var next_slide_html = slides[id].innerHTML;
next_slide.innerHTML = next_slide_html;
next_slide.setAttribute('style', next_slide_data);
}
else {
var next_slide_data = slides[nxt_id].getAttribute('style');
var next_slide_html = slides[nxt_id].innerHTML;
next_slide.innerHTML = next_slide_html;
next_slide.setAttribute('style', next_slide_data);
}
timer = setTimeout(function () {
TweenLite.to(active_slide, 0.5, {marginLeft: '-50%', ease: Power3.easeOut});
setTimeout(function () {
active_slide.setAttribute('style', next_slide.getAttribute('style'));
TweenLite.to(active_slide, 0, {marginLeft: '0%'});
active_slide.innerHTML = next_slide_html;
/* -- RUN LOOP() --- */
setTimeout(function () {
if (type == 'time') {
slideshow(nxt_id, 'time', 7000);
}
else {
slideshow(nxt_id, 'time', time_length);
}
}, 0);
}, 1000);
}, time_length);
}
9.4. Select2 js:lib:select2
https://select2.org/ depends on jQuery
9.4.1. Basic
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script> <link href="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.6-rc.0/css/select2.min.css" rel="stylesheet" /> <script src="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.6-rc.0/js/select2.min.js"></script>
<select class="js-example-basic-single" name="state"> <option value="AL">Alabama</option> ... <option value="WY">Wyoming</option> </select> <script> $(document).ready(function() { $(".js-example-basic-single").select2(); }); </script>
9.4.2. Internal Option data, <optgroup>
Select2 converts each <option> in this JSON
{
"id": "value attribute" || "option text",
"text": "label attribute" || "option text",
"element": HTMLOptionElement
}
//"<optgroup>"
{
"text": "label attribute",
"children": [ option data object, ... ],
"element": HTMLOptGroupElement
}
<select> <optgroup label="Group Name"> <option>Nested option</option> </optgroup> </select>
9.4.3. config:ajax
Supply JSON to Select2
{
"results": [
{
"id": 1,
"text": "Option 1"
},
{
"id": 2,
"text": "Option 2",
"selected": true
},
{
"id": 3,
"text": "Option 3",
"disabled": true
}
]
}
ajax is triggered every time the user types in the search box.
By default, these parameters are sent term :: search term q :: same as term _type :: a request type page ::
e.g. https://api.github.com/search/repositories?term=sel&_type=query&q=sel
$('#mySelect2').select2({
ajax: {
url: 'https://api.github.com/orgs/select2/repos',
data: function (params) {
var query = {
search: params.term,
type: 'public'
}
// Query parameters will be ?search=[term]&type=public
return query;
}
}
});
$('#mySelect2').select2({
ajax: {
url: '/example/api',
processResults: function (data) {
// Tranforms the top-level key of the response object from 'items' to 'results'
return {
results: data.items
};
}
}
});
Pagination
$('#mySelect2').select2({
ajax: {
url: 'https://api.github.com/search/repositories',
data: function (params) {
var query = {
search: params.term,
page: params.page || 1
}
// Query parameters will be ?search=[term]&page=[page]
return query;
}
}
});
If the response has pagination.more prop
{
"results": [
{
"id": 1,
"text": "Option 1"
},
{
"id": 2,
"text": "Option 2"
}
],
"pagination": {
"more": true
}
}
If not, then do it manually. response provides count_filtered that shows the total number.
processResults: function (data, params) {
params.page = params.page || 1;
return {
results: data.results,
pagination: {
more: (params.page * 10) < data.count_filtered
}
};
}
9.4.4. config:data
Load local Javascript array to Select2
var data = [
{
id: 0,
text: 'enhancement'
},
{
id: 1,
text: 'bug'
},
{
id: 2,
text: 'duplicate'
},
{
id: 3,
text: 'invalid'
},
{
id: 4,
text: 'wontfix'
}
];
$(".js-example-data-array").select2({
data: data
})
$(".js-example-data-array-selected").select2({
data: data
})
9.4.5. config:templateResult, Template
function formatState (state) {
if (!state.id) {
return state.text;
}
var baseUrl = "/user/pages/images/flags";
var $state = $(
'<span><img src="' + baseUrl + '/' + state.element.value.toLowerCase() + '.png" class="img-flag" /> ' + state.text + '</span>'
);
return $state;
};
$(".js-example-templating").select2({
templateResult: formatState
});
9.4.6. config:templateSelection, Change display of selected value
function formatState (state) {
if (!state.id) {
return state.text;
}
var baseUrl = "/user/pages/images/flags";
var $state = $(
'<span><img src="' + baseUrl + '/' + state.element.value.toLowerCase() + '.png" class="img-flag" /> ' + state.text + '</span>'
);
return $state;
};
$(".js-example-templating").select2({
templateSelection: formatState
});
9.4.7. config:maximumSelectionLength
$(".js-example-basic-multiple-limit").select2({
maximumSelectionLength: 2
});
9.4.8. config:placeholder
First option has to be empty without any selected option
<select class="js-example-placeholder-single js-states form-control">
<option></option>
</select>
$(".js-example-placeholder-single").select2({
placeholder: "Select a state",
allowClear: true
});
placeholder can be an existing <option>
$('select').select2({
placeholder: {
id: '-1', // the value of the option
text: 'Select an option'
}
});
9.4.9. config:allowClear
Add a x for clearing
$('select').select2({
placeholder: 'This is my placeholder',
allowClear: true
});
9.4.10. config:width
Default is 'resolve' which is to take the style attribute of <select>, if not, fall back to computed element width ('element').
To make sure the field always takes 100% width of parent, use this
'width: '100%'
9.4.11. config:tags
User can type a new option and then select it
<select class="form-control">
<option selected="selected">orange</option>
<option>white</option>
<option>purple</option>
</select>
$(".js-example-tags").select2({
tags: true
});
// multiple
<select class="form-control" multiple="multiple">
<option selected="selected">orange</option>
<option>white</option>
<option selected="selected">purple</option>
</select>
9.4.12. config:tokenSeparators
User can type a space or a comma to add existing or type new option.
$(".js-example-tokenizer").select2({
tags: true,
tokenSeparators: [',', ' ']
})
9.4.13. config:createTag
After a new option is created by typing in config:tags, add properties to this new option Reject creating a new option by returning null
$('select').select2({
createTag: function (params) {
var term = $.trim(params.term);
if (term === '') {
return null;
}
return {
id: term,
text: term,
newTag: true // add additional parameters
}
}
});
9.4.14. config:insertTag
After a new option is created by typing in config:tags, do something about it
$('select').select2({
insertTag: function (data, tag) {
// Insert the tag at the end of the results
data.push(tag);
}
});
9.4.15. config:matcher, config:minimumInputLength, config:minimumResultsForSearch, Search
For single select, a search box is added to search each option. This search can be customized.
It only works for locally supplied data.
function matchCustom(params, data) {
// If there are no search terms, return all of the data
if ($.trim(params.term) === '') {
return data;
}
// Do not display the item if there is no 'text' property
if (typeof data.text === 'undefined') {
return null;
}
// `params.term` should be the term that is used for searching
// `data.text` is the text that is displayed for the data object
if (data.text.indexOf(params.term) > -1) {
var modifiedData = $.extend({}, data, true);
modifiedData.text += ' (matched)';
// You can return modified objects from here
// This includes matching the `children` how you want in nested data sets
return modifiedData;
}
// Return `null` if the term should not be displayed
return null;
}
$(".js-example-matcher").select2({
matcher: matchCustom,
minimumInputLength: 3, // only start searching when the user has input 3 or more characters
minimumResultsForSearch: 20 // at least 20 results must be displayed. -1 or Infinity to permanently hide the search box.
// minimumResultsForSearch: Infinity,
});
9.4.16. Methods, add, select, clear
add
// after $.append, <select> will have the new option
<select id="mySelect2">
var data = {
id: 1,
text: 'Barn owl'
};
var newOption = new Option(data.text, data.id, false, false);
$('#mySelect2').append(newOption).trigger('change');
// create if not exists
// Set the value, creating a new option if necessary
if ($('#mySelect2').find("option[value='" + data.id + "']").length) {
$('#mySelect2').val(data.id).trigger('change');
} else {
// Create a DOM Option and pre-select by default
var newOption = new Option(data.text, data.id, true, true);
// Append it to the select
$('#mySelect2').append(newOption).trigger('change');
}
select
$('#mySelect2').val('1'); // Select the option with a value of '1'
$('#mySelect2').trigger('change'); // Notify any JS components that the value changed
$('#mySelect2').val(['1', '2']);
$('#mySelect2').trigger('change'); // Notify any JS components that the value changed
clear
$('#mySelect2').val(null).trigger('change');
9.4.17. Method, get
// returns a array of objects
$('#mySelect2').select2('data');
$('#mySelect2').find(':selected');
// get value
$('#mySelect2').val();
// pure javascript will not work
// document.querySelector('#mySelect2').value;
$('#mySelect2').select2({
// ...
templateSelection: function (data, container) {
// Add custom attributes to the <option> tag for the selected option
$(data.element).attr('data-custom-attribute', data.customValue);
return data.text;
}
});
// Retrieve custom attribute value of the first selected element
$('#mySelect2').find(':selected').data('custom-attribute');
9.4.18. Method, open|close dropdown, if initialized, destroy
$('#mySelect2').select2('open');
$('#mySelect2').select2('close');
if ($('#mySelect2').hasClass("select2-hidden-accessible")) {
// Select2 has been initialized
}
$('#mySelect2').select2('destroy');
Custom events need to be unbinded after destroy
// Set up a Select2 control
$('#example').select2();
// Bind an event
$('#example').on('select2:select', function (e) {
console.log('select event');
});
// Destroy Select2
$('#example').select2('destroy');
// Unbind the event
$('#example').off('select2:select');
9.4.19. Example: Destroy and Initiate
<button class="js-programmatic-destroy button">Destroy</button> <button class="js-programmatic-init button">Re-initialize</button> <script> $(".js-programmatic-destroy").on("click", function () { $example.select2("destroy"); }); $(".js-programmatic-init").on("click", function () { $example.select2(); }); </script>
9.4.20. Add down arrow to multiple <select>
By default, multiple <select> doesn't have a down arrow. The solution is to copy .select2-selection--single to .select2-selection--multiple
.select2-container--default .select2-selection--multiple .select2-selection__rendered{ padding-left:8px; padding-right:20px; } .select2-container--default .select2-selection--multiple .select2-selection__clear{ font-size:13px; color:#666; margin:0; line-height:26px; } .select2-selection--multiple:after{ content:""; position:absolute; right:7px; top:12px; width:0; height:0; border-left: 4px solid transparent; border-right: 4px solid transparent; border-top: 5px solid #888; }
9.4.21. Events
change :: general Javascript select change change.select2 :: only for Select2 <select> change select2:selecting :: Triggered before a result is selected. This event can be prevented. select2:select :: Triggered whenever a result is selected. select2:selecting is fired before this and can be prevented.
$('#mySelect2').on('select2:select', function (e) {
var data = e.params.data;
console.log(data);
});
Trigger an event
var data = {
"id": 1,
"text": "Tyto alba",
"genus": "Tyto",
"species": "alba"
};
$('#mySelect2').trigger({
type: 'select2:select',
params: {
data: data
}
});
$('#mySelect2').trigger('change.select2'); // Notify only Select2 of changes
9.5. Owl Carousel
https://github.com/OwlCarousel2/OwlCarousel2
Bootstrap's carousel can only show one item like a slideshow. Owl can have multiple items.
Example: Owl Carousel + Bootstrap Modal + Bootstrap Carousel
9.6. Fine Uploader
https://github.com/FineUploader/fine-uploader No dependency
9.7. Dropzone
https://github.com/enyo/dropzone/
No dependency
9.7.1. Basics
Download the dist folder
<script src="/dist/dropzone.js"></script> <link rel="stylesheet" href="/dist/dropzone.css"> <!-- /dist/min/* for production -->
Dropzone will be available as window.Dropzone and jQuery.fn.Dropzone (if jQuery is loaded)
Create Dropzone without javascript (auto discover), Dropzone.options
<form action="/file-upload" class="dropzone" id="my-awesome-dropzone">
<!-- hidden input will be submitted -->
<!-- <input type="hidden" name="addtionaldata" value="1" /> -->
<div class="fallback">
<input name="file" type="file" multiple />
</div>
</form>
<!-- because it's auto discovered, you may need to setup the config -->
// "myAwesomeDropzone" is the camelized version of the HTML element's ID
Dropzone.options.myAwesomeDropzone = {
paramName: "file", // The name that will be used to transfer the file
maxFilesize: 2, // MB
accept: function(file, done) {
if (file.name == "justinbieber.jpg") {
done("Naha, you don't.");
}
else { done(); }
}
};
Create with javascript, must have url option.
// Prevent Dropzone from auto discovering this element:
Dropzone.options.myAwesomeDropzone = false;
// This is useful when you want to create the
// Dropzone programmatically later
// Disable auto discover for all elements:
Dropzone.autoDiscover = false;
// Dropzone class:
var myDropzone = new Dropzone("div#myId", { url: "/file/post"});
// jQuery
$("div#myId").dropzone({ url: "/file/post" });
The file will be posted in backend as
<input type="file" name="file" />
To make the whole page droppable
<!--
Don't forget to give this container the dropzone-previews class
so the previews are formatted correctly.
-->
<div id="previews" class="dropzone-previews"></div>
<button id="clickable">Click me to select files</button>
var myDropzone = new Dropzone(document.body, {
previewsContainer: ".dropzone-previews", // the element must have
// You probably don't want the whole body
// to be clickable to select files
clickable: false
});
// or
<script>
new Dropzone(document.body, { // Make the whole body a dropzone
url: "/upload/url", // Set the url
previewsContainer: "#previews", // Define the container to display the previews
clickable: "#clickable" // Define the element that should be used as click trigger to select files.
});
</script>
9.7.2. option:function:accept(file, done)
Invoke done() without arguments to accept and process the file. Invoke done() with an error message to reject the file and show the message. It won't be invoked if the file is too big or mime type doesn't match.
accept: function(file, done) {
if (file.name == "justinbieber.jpg") {
done("Naha, you don't.");
}
else { done(); }
}
9.7.3. option:previewsContainer null|string
HTMLElement or a CSS selector. The element should have the .dropzone-previews or dropzone class.
9.7.4. option:autoProcessQueue
False :: files will be added to the queue but the queue will not be processed automatically. Call myDropzone.processQueue() to process the queue.
9.7.5. option:previewTemplate String, option:dict*
Default template dropzone:default template
<div class="dz-preview dz-file-preview">
<div class="dz-details">
<div class="dz-filename"><span data-dz-name></span></div>
<div class="dz-size" data-dz-size></div>
<!-- img's alt and src will change -->
<img data-dz-thumbnail />
</div>
<div class="dz-progress"><span class="dz-upload" data-dz-uploadprogress><!-- style.width from 0% to 100% --></span></div>
<div class="dz-success-mark"><span>✔</span></div>
<div class="dz-error-mark"><span>✘</span></div>
<div class="dz-error-message"><span data-dz-errormessage></span></div>
<!-- <a class="dz-remove" href="javascript:undefined" data-dz-remove>Remove file</a> -->
</div>
<!-- Another preview -->
The default template might change from version to version. But console.log(file.previewElement) in any event will output.
.dz-preview gets .dz-processing when the file gets processed, .dz-success when it got uploaded and .dz-error
Make sure custom template has these attributes
- data-dz-name
- data-dz-size
- data-dz-thumbnail
- data-dz-uploadprogress
- data-dz-errormessage
Add a remove button (data-dz-remove)
<img src="removebutton.png" alt="Click me to remove the file." data-dz-remove />
9.7.6. option:addRemoveLInks null:true
Wording: dictCancelUpload, dictCancelUploadConfirmation and dictRemoveFile dropzone:default template
9.7.7. option:translation
9.7.7.1. option:dictCancelUpload, dictCancelUploadConfirmation, dictRemoveFile
9.7.8. option:maxFiles int
Dropzone.options.myAwesomeDropzone = {
maxFiles: 1,
accept: function(file, done) {
console.log("uploaded");
done();
},
init: function() {
this.on("maxfilesexceeded", function(file){
alert("No more files please!");
this.removeFile(file);
});
}
};
9.7.9. option:maxFilesize int (MB)
event:maxfilesexceeded is called and the whole dropzone element gets the class .dz-max-files-reached
9.7.10. option:event:init, add events
It's the best place to add custom events if options are used to setup the dropzone
// The recommended way from within the init configuration:
Dropzone.options.myAwesomeDropzone = {
init: function() {
this.on("addedfile", function(file) { alert("Added file."); });
}
};
Use this to attach events if dropzone is defined programmatically
// This example uses jQuery so it creates the Dropzone, only when the DOM has
// loaded.
// Disabling autoDiscover, otherwise Dropzone will try to attach twice.
Dropzone.autoDiscover = false;
// or disable for specific dropzone:
// Dropzone.options.myDropzone = false;
$(function() {
// Now that the DOM is fully loaded, create the dropzone, and setup the
// event listeners
var myDropzone = new Dropzone("#my-dropzone");
myDropzone.on("addedfile", function(file) {
/* Maybe display some more file information on your page */
});
})
9.7.11. option:clickable null|true|false|an html element|a CSS selector|array of html
All of those elements will trigger an upload when clicked
9.7.12. option:acceptedFiles string of list
comma separated list of mime types of file extensions
image/*,applicaton/pdf,.psd
9.7.13. option:maxThumbnailFilesize (MB), thumbnailWidth, thumbnailHeight, thumbnailMethod
thumbnailMethod :: 'contain' or 'crop' when thumbnailWidth and thumbnailHeight are both defined!
9.7.14. event:addedfile
// Disabling autoDiscover, otherwise Dropzone will try to attach twice.
Dropzone.autoDiscover = false;
// or disable for specific dropzone:
// Dropzone.options.myDropzone = false;
$(function() {
// Now that the DOM is fully loaded, create the dropzone, and setup the
// event listeners
var myDropzone = new Dropzone("#my-dropzone");
myDropzone.on("addedfile", function(file) {
/* Maybe display some more file information on your page */
});
})
9.7.15. event:removedfile(file)
9.7.16. event:thumbnail(file, dataUrl)
9.7.17. event:processing(file)
when a file gets processed
9.7.18. event:uploadprogress(file, progress, bytesSent)
progress :: int from 0 to 100
9.7.19. event:error(file)
9.7.20. event:success(file, responseText)
Display something after a file is uploaded.
myDropzone.files has been appended.
Dropzone.options.myDropzone = {
init: function() {
this.on("success", function(file, responseText) {
// Handle the responseText here. For example, add the text to the preview element:
file.previewTemplate.appendChild(document.createTextNode(responseText));
});
}
};
Display thumbnail that is created on server after server processes the uploaded image { "imageUrl": "http://my.image/file.jpg" }
Dropzone.options.myDropzone = {
init: function() {
this.on("success", function(file, serverResponse) {
// Called after the file successfully uploaded.
// If the image is already a thumbnail:
this.emit('thumbnail', file, serverResponse.imageUrl);
// If it needs resizing:
this.createThumbnailFromUrl(file, serverResponse.imageUrl);
});
}
};
9.7.21. event:complete(file)
9.7.22. event:canceled(file)
9.7.23. event:maxfilesexceeded(file)
9.7.24. event:maxfilesreached(file)
the number of files accepted reaches the maxFiles limit
9.7.26. Show error returned by server
Just return HTTP status code in the range of 400-500. If response Content-Type is text/plain, the text will be returned as error message. If it's application/json, dropzone uses the error property {"error": "File could not be saved."}
9.7.27. Show server response
See event:success(file, responseText)
9.7.28. Show files already stored on server
// Create the mock file:
var mockFile = { name: "Filename", size: 12345 };
// Call the default addedfile event handler
myDropzone.emit("addedfile", mockFile);
// And optionally show the thumbnail of the file:
myDropzone.emit("thumbnail", mockFile, "/image/url");
// Or if the file on your server is not yet in the right
// size, you can let Dropzone download and resize it
// callback and crossOrigin are optional.
myDropzone.createThumbnailFromUrl(file, imageUrl, callback, crossOrigin);
// Make sure that there is no progress bar, etc...
myDropzone.emit("complete", mockFile);
myDropzone.files.push(mockFile);
// If you use the maxFiles option, make sure you adjust it to the
// correct amount:
var existingFileCount = 1; // The number of files already uploaded
myDropzone.options.maxFiles = myDropzone.options.maxFiles - existingFileCount;
9.7.29. Sortable
$(function() {
$("#slideshow-pictures").sortable({
items:'.dz-preview',
cursor: 'move',
opacity: 0.5,
containment: '#slideshow-pictures',
//distance: 20,
tolerance: 'pointer',
update: function (event, ui) {
var queue = myDropzone.files;
var newQueue = [];
$('#slideshow-pictures .dz-preview .dz-filename [data-dz-name]').each(function (count, el) {
var name = el.innerHTML;
queue.forEach(function(file) {
if (file.name === name) {
newQueue.push(file);
}
});
});
myDropzone.files = newQueue;
}
});
});
Dropzone.options.slideshowPictures = false;
Dropzone.autoDiscover = false;
var myDropzone = new Dropzone("#slideshow-pictures", {...});
9.8. modernizr
It usually adds a class in <html> when a certain feature is supported
// <html class="no-cssgradients">
// <html class="cssgradients">
.no-cssgradients .header {
background: url("images/glossybutton.png");
}
.cssgradients .header {
background-image: linear-gradient(cornflowerblue, rebeccapurple);
}
Config
{
"classPrefix": "foo-",
"feature-detects": ["dom/hidden"]
}
9.8.1. Feature
Modernizr.feature-name is true or false, true add .classPrefix-feature-name, false add .classPrefix-no-feature-name
if (Modernizr.mediaqueries) {
// supported
} else {
// not-supported
}
9.8.1.1. flexbox, flexboxlegacy, flexboxtweener, flexwrap modernizr:flexbox
.flexbox.no-flexwrap .row {
display:-webkit-flex-wrap: wrap;
}
Bootstrap 4 fallback flex
Lili.cssFixFlex = function() {
if (!Modernizr.testProp('display', 'flex') && !Modernizr.testProp('display', '-ms-flexbox') && Modernizr.testProp('display', '-webkit-box') && Modernizr.testProp('display', '-webkit-flex')) {
// display:flex and -ms-flexbox are not supported but -webkit-box and -webkit-flex are supported
// Bootstrap uses display:-webkit-box only
// add class to html tag
$('html').addClass('bootstrap-ios-fix-row');
}
else if (!Modernizr.testProp('display', 'flex') && !Modernizr.testProp('display', '-ms-flexbox') && Modernizr.testProp('display', '-webkit-box') && !Modernizr.testProp('display', '-webkit-flex')) {
// flex is completely not supported
$('html').addClass('bootstrap-fix-row');
}
}
$(function() {
Lili.cssFixFlex();
})
.bootstrap-ios-fix-row .row {
display:-webkit-flex;
-webkit-flex-wrap:wrap;
}
.bootstrap-fix-row .row {
display:inline-block;
}
9.8.1.2. touchevents
9.8.1.3. videoautoplay
This feature detection is not reliable..
9.8.1.4. webp modernizr:webp
Modernizr.webp, Modernizr.webp.lossless, Modernizr.webp.alpha and Modernizr.webp.animation
9.8.2. Option
9.8.2.1. .mq(mq)
var query = Modernizr.mq('(min-width: 900px)');
if (query) {
// the browser window is larger than 900px
}
Modernizr.mq('only all'); // true if MQ are supported, false if not
9.8.3. API
9.8.3.1. Modernizr.on(feature, cb)
Modernizr.on('flash', function( result ) {
if (result) {
// the browser has flash
} else {
// the browser does not have flash
}
});
9.8.3.2. .addTest('customFeatureName', cb)
Modernizr.addTest('itsTuesday', function() {
var d = new Date();
return d.getDay() === 2;
});
Modernizr.addTest('hasJquery', 'jQuery' in window);
// combine
var detects = {
'hasjquery': 'jQuery' in window,
'itstuesday': function() {
var d = new Date();
return d.getDay() === 2;
}
}
Modernizr.addTest(detects);
9.8.3.3. .atRule(prop)
var keyframes = Modernizr.atRule('@keyframes');
if (keyframes) {
// keyframes are supported
// could be `@-webkit-keyframes` or `@keyframes`
} else {
// keyframes === `false`
}
9.8.3.4. ._domPrefixes, ._prefixes
._prefixes return kebab-case properties
var rule = Modernizr._prefixes.join('transform: rotate(20deg); ');
rule === 'transform: rotate(20deg); webkit-transform: rotate(20deg); moz-transform: rotate(20deg); o-transform: rotate(20deg); ms-transform: rotate(20deg);'
Modernizr._domPrefixes = [ "Moz", "O", "ms", "Webkit" ];
9.8.3.5. .hasEvent(eventName, [element])
9.8.3.6. .prefixedCSS(prop), .prefixed(prop, [obj], [elem])
Modernizr.prefixedCSS('transition') // '-moz-transition' in old Firefox
9.8.3.7. .prefixedCSSValue(prop,value)
9.8.3.8. .testAllProps(prop,[value], [skipValueTest]), .testProp(prop,[value], [useValue])
Whether a given CSS property, in some prefixed form, is supported by the browser.
testAllProps('boxSizing') // true
testAllProps('display', 'block') // true
testAllProps('display', 'penguin') // false
.testProp only tests against non-prefix version.
9.9. UAParser.js
Browser detect, engine, OS, CPU and device type/model from userAgent string. Git
<script src="https://cdn.jsdelivr.net/npm/ua-parser-js@0/dist/ua-parser.min.js"></script>
var parser = new UAParser();
var r = parser.getResult(); // { ua: '', browser: {}, cpu: {}, device: {}, engine: {}, os: {} }
console.log(r.browser); // {name: "Chromium", version: "15.0.874.106"}
// browser.name
// Amaya, Android Browser, Arora, Avant, Baidu, Blazer, Bolt, Bowser, Camino, Chimera,
Chrome [WebView], Chromium, Comodo Dragon, Conkeror, Dillo, Dolphin, Doris, Edge,
Epiphany, Fennec, Firebird, Firefox, Flock, GoBrowser, iCab, ICE Browser, IceApe,
IceCat, IceDragon, Iceweasel, IE[Mobile], Iron, Jasmine, K-Meleon, Konqueror, Kindle,
Links, Lunascape, Lynx, Maemo, Maxthon, Midori, Minimo, MIUI Browser, [Mobile] Safari,
Mosaic, Mozilla, Netfront, Netscape, NetSurf, Nokia, OmniWeb, Opera [Mini/Mobi/Tablet],
PhantomJS, Phoenix, Polaris, QQBrowser, QQBrowserLite, Quark, RockMelt, Silk, Skyfire,
SeaMonkey, Sleipnir, SlimBrowser, Swiftfox, Tizen, UCBrowser, Vivaldi, w3m, Waterfox,
WeChat, Yandex
r.device // { model: '', type: '', vendor: '' }
// model
// console, mobile, tablet, smarttv, wearable, embedded
9.10. flowpaper
Turn PDF into HTML5. https://flowpaper.com/download/
9.11. reveal.js
9.11.1. Plugins
9.11.1.1. reveal.js-menu
Git clone and copy the whole folder
Reveal.initialize({
// ...
dependencies: [
// ...
{ src: 'plugin/reveal.js-menu/menu.js' }
]
});
Add id="theme" if theme is used
<link rel="stylesheet" href="css/theme/black.css" id="theme">
Add data-menu-title to section, if not, the first .menu-title is used and it doesn't have to be displayed. Or change option:titleSelector. If still no title and option:hideMissingTitles is set to false, then the slide will not be included in menu
Add custom item (panel) in menu top, default is slides and close button.
Reveal.initialize({
// ...
menu: {
// ...
custom: [
{ title: 'Links', icon: '<i class="fa fa-external-link">', src: 'links.html' },
{ title: 'About', icon: '<i class="fa fa-info">', content: '<p>This slidedeck is created with reveal.js</p>' }
]
}
});
- src
- when clicked, the external source is loaded into the menu content section.
- content
- any html but you can add your own menu items
<h1>Links</h1>
<ul class="slide-menu-items">
<li class="slide-menu-item"><a href="#/transitions">Transitions</a></li>
<li class="slide-menu-item"><a href="#/13">Code highlighting</a></li>
</ul>
// you can also link to any where
<h1>External Links</h1>
<ul class="slide-menu-items">
<li class="slide-menu-item"><a href="https://github.com/denehyg/reveal.js-menu">Reveal.js-menu</a></li>
<li class="slide-menu-item"><a href="https://github.com/hakimel/reveal.js">Reveal.js</a></li>
</ul>
Options and their default
Reveal.initialize({
// ...
menu: {
side: 'left', // or 'right',
markers: true, // Add icon to menu title
},
});
- titleSelector: 'h1, h2, h3, h4, h5, h6'
- e.g. 'p.lead'
- (no term)
- hideMissingTitles: false
9.13. BabylonJS
- Install Blender
- https://www.blender.org/
- (no term)
- File > User Preferences > Add-ons > search dxf and enable Export and Import AutoCAD DXF Format
- (no term)
- Or import "dxf ascii release 14 or under"
- Install addon to export from .blend to .babylon
- http://doc.babylonjs.com/resources/blender
- (no term)
- Export as .babylon
- (no term)
- Get free 3d .obj, .fbx, .3ds files from Free3d.com or Clara.io
- (no term)
- Tutorial: https://github.com/mpwassler/3dproductview
- (no term)
- Callback in runRenderLoop is run 60 times per second to draw a frame
9.13.1. Shapes
https://doc.babylonjs.com/babylon101/discover_basic_elements
var shape = BABYLON.MeshBuilder.Create[ShapeName](name, options, scene);
- box, Sphere, Plane, Ground
9.14. Angular
9.14.1. Who use Angular?
9.14.2. 1.x AngularJS
9.14.2.1. Structure
https://ajax.googleapis.com/ajax/libs/angularjs/1.4.7/angular.js jquery common/common.module.js // angular.module('ajsNgComponents', []); common/config.js ng:config common/filters.js ng:filter common/fitlerableCategories.js ng:factory
9.14.2.2. Calling Order
app.config() app.run() directive's compile functions (if exist) app.controller() directive's link functions
9.14.2.3. Config ng:config
angular .module('ajsNgComponents') .config(locatgionProviderConfig) // .config is to run a Provider function // ng:provider .value('layout','boxed_fullwidth') .value('layoutClass', { boxed_fullwidth: 'col-lg-3 col-md-4 col-sm-6 col-xs-12', boxed_left: 'col-lg-4 col-md-6 col-sm-6 col-xs-12', boxed_right: 'col-lg-4 col-md-6 col-sm-6 col-xs-12' }); // Enable html5mode locatgionProviderConfig.$inject = ['$locationProvider']; function locatgionProviderConfig($locationProvider) { $locationProvider.html5Mode({ enabled: true, requireBase: false }); }
9.14.2.4. Run
Run injection is a bit different
app.run(ajsRun);
// Not app.run('ajsRun', ajsRun);
ajsRun.$inject = ['$rootScope'];
function ajsRun($rootScope) {
}
9.14.2.5. Filter ng:filter
angular .module('ajsNgComponents') .filter('singleListingFilter', singleListingFilter) .filter('item_category_value', item_category_value) .filter('ignore_empty_value', ignore_empty_value) .filter('translate', translate); // $scope can't be injected to filter // Instead, pass scope.varname as a param into filter. // Or inject $rootScope // In view /* <tr ng-repeat="detail in details | ignore_empty_value:aListing" ng-init="detail = aListing.post_meta.options[detailKey]"> <td ng-bind="(detail.name | translate:detail)"></td> <td ng-bind="(detail.value | translate)"></td> </tr> */ function singleListingFilter() { return function(items,var1) { var r = []; for (var i= 0, len=items.length; i< len; i++) { if (items[i].id == var1){ r.push(items[i]); break; } } return r; }; } // <td ng-bind="((x.options | item_category_value:detail:lwp_options) | translate)"></td> function item_category_value() { return function(options, category, lwp_options) { var value = ""; return value; }; } function ignore_empty_value() { return function(items, aListing) { var r = []; angular.forEach(items, function(item) { if (aListing.post_meta.options[item].value) { r.push(item); } }); return r; }; } translate.$inject = ['$rootScope']; function translate($rootScope) { return function(word, languages) { var r = word; if (typeof $rootScope.lang !== "undefined") { if ( typeof languages !== "undefined" && typeof languages['lang'] !== "undefined" && typeof languages['lang'][$rootScope.lang] !== "undefined") { r = languages['lang'][$rootScope.lang]; } else { r = $rootScope.translate_text(word); } } return r; }; }
9.14.2.6. Factory ng:factory
Factory is better than app.value();
(function() { 'use strict'; angular .module('ajsNgComponents') .factory('filterableCategories', filterableCategories) .factory('clientId', clientIdFactory) .factory('apiToken', apiTokenFactory); // Inject $rootScope so that any method defined here can use filterableCategories.$inject = ['$rootScope']; function filterableCategories($rootScope) { return { getFilters: getFilters, getListingsFilterObjByURL: getListingsFilterObjByURL, getSelectedFiltersObjByURL: getSelectedFiltersObjByURL, getSearchInventoryDropdown: getSearchInventoryDropdown }; function getFilters(filterable_categories, lwp_options) { // $rootScope can be used here var r = []; return r; } } function clientIdFactory() { return '123'; } apiTokenFactory.$inject = ['clientId']; function apiTokenFactory(clientId) { var encrypt = function(data1, data2) { return data1+data2; } var secret = window.localStorage.getItem('secret'); var apiToken = encrypt(clientId, secret); return apiToken; } })();
9.14.2.7. Service ng:service
app.service('unicornLauncher', UnicornLauncherService); UnicornLauncherService.$inject = ['apiToken']; function UnicornLauncherService(apiToken) { this.launchedCount = 0; this.launch = function () { // Use apiToken to do something this.launchedCount++; } }
9.14.2.8. Provider ng:provider
Provider is a service and it's the only thing can be inserted to .config
Instantiation happens in config
app.config(myProviderConfig); myProviderConfig.$inject = ['myProviderProvider']; function myProviderConfig(myProviderProvider){ myProviderProvider.thingFromConfig = 'This was set in config'; });
Define myProvider
app.provider('myProvider', function(){ // app.config() can only access the next 2 lines this._artist = ''; this.thingFromConfig = ''; this.$get = function(){ var that = this; return { getArtist: function(){ return that._artist; }, thingOnConfig: that.thingFromConfig, unicornLauncher: newUnicornLauncher } newUnicornLauncher.$inject = ['apiToken']; function newUnicornLaunder(apiToken) { return new UnicornLauncherService(apiToken); } } });
app.controller('myProvider', function($scope, myProvider){ $scope.artist = myProvider.getArtist(); $scope.data.thingFromConfig = myProvider.thingOnConfig; });
9.14.2.9. Change URL without reloading page
First enable ng:html5mode
inject $location and $window
myNewUrl = '/path/?p=1&q=2'; $location.url(myNewUrl); $location.replace(); $window.history.pushState(null, 'any', $location.absUrl());
9.14.3. 2.x Angular
9.14.3.1. QuickStart seed
- Install
Local dev env which is the same as Online playground https://angular.io/guide/setup
git clone https://github.com/angular/quickstart.git quickstart cd quickstart npm install npm start # Default port is localhost:3000
Backup
.gitignoreAnd remove git-related artifacts
# OS/X xargs rm -rf < non-essential-files.osx.txt rm src/app/*.spec*.ts rm non-essential-files.osx.txt # Windows cmd for /f %i in (non-essential-files.txt) do del %i /F /S /Q rd .git /s /q rd e2e /s /q
- .gitignore
.idea node_modules jspm_packages npm-debug.log debug.log src/**/*.js !src/systemjs.config.extras.js !src/systemjs.config.js !src/systemjs-angular-loader.js *.js.map e2e/**/*.js e2e/**/*.js.map _test-output _temp
- Files
https://angular.io/guide/setup-systemjs-anatomy
src/main.ts It's loaded in index.html as in
System.import('main.js').catch(function(err){ console.error(err); });Refer to ng:bootstrapsrc/systemjs.config.js It's loaded in index.html. Tells systemJS module loader where to find modules referenced in JS
importstatements. e.g.import { Component } from '@angular/core'src/tsconfig.json :: how to transpile TypeScript files into JS files
src/app/app.module.ts :: ng:root module src/app/app.component.ts :: tree of components ng:component
package.json tslint.json :: Inspect TypeScript code and complains when there's violation bs-config.json :: lite-server BrowserSync setting
9.14.3.2. SystemJS vs Webpack
Quickstart uses SystemJS to dynamically load files on demand on the client side. It's also called lazy load. Angular CLI uses Webpack to prepare files. Also does file minifying, transpilation (e.g. from SASS to CSS).
9.14.3.3. package.json
- devDependencies
"devDependencies": { "concurrently": "^3.2.0", "lite-server": "^2.2.2", // static file server "typescript": "~2.1.0", // tsc TypeScript compiler "canonical-path": "0.0.2", "tslint": "^3.15.1", "lodash": "^4.16.4", "jasmine-core": "~2.4.1", "karma": "^1.3.0", "karma-chrome-launcher": "^2.0.0", "karma-cli": "^1.0.1", "karma-jasmine": "^1.0.2", "karma-jasmine-html-reporter": "^0.2.2", "protractor": "~4.0.14", "rimraf": "^2.5.4", "@types/node": "^6.0.46", "@types/jasmine": "2.5.36" }
- dependencies
"dependencies": { "@angular/common": "~4.0.0", // common services, pipes, and directives provided by Angular "@angular/compiler": "~4.0.0", // Angular Template Compiler "@angular/core": "~4.0.0", // e.g. all metadata decorators, Component, Directive, dependency injection and component lifecycle hooks "@angular/forms": "~4.0.0", "@angular/http": "~4.0.0", // Angular's HTTP client. "@angular/platform-browser": "~4.0.0", // DOM and browser related. Native directives, pipes, etc. Help render into DOM. "@angular/platform-browser-dynamic": "~4.0.0", // Include Providers and a bootstrap method for apps that compile templates on the client. // Don't use offline compilation, use this for bootstrapping during dev and for bootstrapping plunker samples "@angular/router": "~4.0.0", // Component router "angular-in-memory-web-api": "~0.3.0", // Angular simulates a remote server's web api without requiring an actual server or real HTTP calls. "systemjs": "0.19.40", // A dynamic module loader compatible with ES2015 module specs. Alternative webpack "core-js": "^2.4.1", // polyfill "rxjs": "5.0.1", // polyfill "zone.js": "^0.8.4" // polyfill }
- scripts
"scripts": { "build": "tsc -p src/", "build:watch": "tsc -p src/ -w", "build:e2e": "tsc -p e2e/", "serve": "lite-server -c=bs-config.json", "serve:e2e": "lite-server -c=bs-config.e2e.json", "prestart": "npm run build", "start": "concurrently \"npm run build:watch\" \"npm run serve\"", "pree2e": "npm run build:e2e", "e2e": "concurrently \"npm run serve:e2e\" \"npm run protractor\" --kill-others --success first", "preprotractor": "webdriver-manager update", "protractor": "protractor protractor.config.js", "pretest": "npm run build", "test": "concurrently \"npm run build:watch\" \"karma start karma.conf.js\"", "pretest:once": "npm run build", "test:once": "karma start karma.conf.js --single-run", "lint": "tslint ./src/**/*.ts -t verbose" }
9.14.3.4. tsconfig.json, d.ts, lib.d.ts
https://angular.io/guide/typescript-configuration http://www.typescriptlang.org/docs/handbook/tsconfig-json.html
Quickstart src/tsconfig.json
{
"compilerOptions": {
"target": "es5",
"module": "commonjs",
"moduleResolution": "node",
"sourceMap": true,
"emitDecoratorMetadata": true,
"experimentalDecorators": true,
"lib": [ "es2015", "dom" ],
"noImplicitAny": true,
"suppressImplicitAnyIndexErrors": true
}
}
Angular CLI src/tsconfig.app.json extends ./tsconfig.json. See ./.angular-cli.json
d.ts file is a TypeScript Declaration file, which tells the compiler what libraries you load.
Libraries, such as jQuery and Angular extend the JavaScript environment with features and syntax.
node_modules/@angular/core/ folder contains several d.ts files. You don't need to do anything to get these typing files for Angular library packages.
lib.XXX.d.ts
Typescript includes many lib.XXX.d.ts files for you to load ambient declarations.
In tsconfig.json for Quickstart, there's a line to load lib.es2015.d.ts and lib.dom.d.ts
Based on the --target, TypeScript adds other declarations automatically like Promose if the target is es6
"lib": ["es2015", "dom"]
Since many libraries, e.g. jQuery, Jasmine and Lodash, do not include d.ts files in npm packages.
npm install these scoped packages @types/* and Typescript automatically recognizes them.
Quickstart dependencies have 2: @types/node @types/jasmine
9.14.3.5. lite-server npm:lite-server
9.14.3.6. CLI
- Install & Update
# Install or update npm install -g @angular/cli
Update :: update both the global package and your project's local package.
# Global package: npm uninstall -g @angular/cli npm cache clean npm install -g @angular/cli@latest Local project package: rm -rf node_modules dist # use this in Windows Command Prompt # rmdir /S/Q node_modules dist # use this in Windows PowerShell # rm -r -fo node_modules,dist npm install --save-dev @angular/cli@latest npm install
- Generate Component, Directive, Pipes and Services
CLI adds reference in app.module.ts automatically.
# ng generate ng g component my-new-component ng g component ../newer-cmp ng g component feature/new-cmp ng g directive new-directive ng g pipe new-pipe ng g service new-service ng g class new-class ng g guard new-guard ng g interface new-interface ng g enum new-enum ng g module new-module
- New project
ng new my-project cd my-project ng serve
src/index.html doesn't include any javascript files while quickstart includes some.
- build
ng build --prod:: Delete and add files to ./distAll commands that build or serve your project,
ng build/serve/e2e, will delete the output directory (dist/ by default). This can be disabled via the--no-delete-output-path(or--delete-output-path=false) flag. - eject ng:cli:webpack
Do this after you have stopped
ng serve. This will copy the webpack configuration and modify .angular-cli.json and package.json. Dogit status --ignoredto see the webpack configurations. Afterng eject, ng commands will not work. See changes in package.json. Such as: instead ofng serve, donpm run build & npm run startYou can now modify webpack configuration with real effects.ng eject --prodproduction environment. It includes UglifyJsPlugin npm:webpackBundle files are main.ts, polyfills.ts and styles.css
9.14.3.7. Polyfills
<script src="node_modules/core-js/client/shim.min.js"></script>
9.14.3.8. Bootstrap ng:bootstrap
Create a browser platform for dynamic compilation (JIT) and bootstraps the AppModule by referring to the root module app.module.ts Refer to ng:root module
import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';
import { AppModule } from './app/app.module';
platformBrowserDynamic().bootstrapModule(AppModule);
9.14.3.9. Root Module ng:root module
import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { AppComponent } from './app.component';
// import { FormsModule } from '@angular/forms';
// This decorator makes AppModule as an Angular module class (NgModule class).
// a metadata object to tell Angular how to compile and launch the app
@NgModule({
// Only put NgModule classes in imports array
imports: [ BrowserModule
// , FormsModule
],
// Only declarables - components, directives and pipes - belong in declarations array. Not NgModule classes
declarations: [ AppComponent ],
// Create components listed in the bootstrap array and insert each one into browser DOM
// It's common to only include one tree of components, which is one file: app.component.ts
bootstrap: [ AppComponent ]
})
export class AppModule { }
9.14.3.10. ngModule
9.14.3.11. Component ng:component
- Create a new html element. It's also a directive with a template
- Refer to ng:api:component
src/app/app.component.ts
import { Component } from '@angular/core'; // Decorator @Component({ selector: 'my-app', template: `<h1>Hello {{name}}</h1>`, inputs: ['inputVar1'], // Use this or @Input. Refer to ng:input output outputs: ['outputVar1'], // Use this or @Output. Refer to ng:input output }) export class AppComponent { name = 'Angular'; }
9.14.3.12. Template
- Files
src/app/[component-name]/[component-name].component.html src/app/[component-name]/[component-name].component.css src/app/[component-name]/[component-name].component.ts
src/app/heroes/heroes.component.ts Use component-relative URLs, prefixed with
./@Component({ selector: 'toh-heroes', templateUrl: './heroes.component.html', // template or templateUrl, can't be both styleUrls: ['./heroes.component.css'] // each css file is independent! Don't worry about CSS conflicts // , styles: [ ".btn {background-color: green;}", ".btn:hover {background-color: pink;}"] }) export class HeroesComponent implements OnInit { /* name = 'abc'; artists: string[]; // could be any */ /* or name: string; constructor() { this.name = 'abc'; this.artists = ["1", "2"] } */ heroes: Observable<Hero[]>; selectedHero: Hero; onClick(e) { // event // console.log(e); this.name = e.target.innerHTML; } addArtist(v) { if (v!=='') { this.artists.push({ name: v, school: 'abc' }); } } constructor(private heroService: HeroService) { } ngOnInit() { this.heroes = this.heroService.getHeroes(); } }src/app/heroes/heroes.component.html
<div> <h2>My Heroes</h2> <ul class="heroes"> <li *ngFor="let hero of heroes | async" (click)="selectedHero=hero"> <span class="badge">{{hero.id}}</span> {{hero.name}} </li> </ul> <div *ngIf="selectedHero"> <h2>{{selectedHero.name | uppercase}} is my hero</h2> </div> </div> {{ }}
Nonsupported in
{{ }}- Assignments
- Newing up variables
- Chaining expressions
- Incrementing/decrementing
A function can be run
<div>{{ wasWatched() }}</div> export class MediaItemComponent { wasWatched() { return true; } }- Styles
Module bundler like Webpack can load sytles from external files at build time.
styles: [require('my.component.css')]Use
:hostto target the host element (one element):host-context()looks for a CSS class in any ancestor of the component host element, up to the document root. The :host-context() selector is useful when combined with another selector.This applies a background-color style to all <h2> elements inside the component, only if some ancestor element has the CSS class theme-light.
:host-context(.theme-light) h2 { background-color: #eef; }/deep/or>>>works to any depth of nested components, and it applies to both the view children and content children of the component.This targets all <h3> elements, from the host element down through this component to all of its child elements in the DOM.
:host /deep/ h3 { font-style: italic; }Use
/deep/and>>>selectors only with emulated view encapsulation. Emulated is the default and most commonly used view encapsulation. Refer to ng:view encapsulationUse
<link>in template, path is relative to the root.@Component({ selector: 'hero-team', template: ` <link rel="stylesheet" href="app/hero-team.component.css"> <h3>Team</h3> <ul> <li *ngFor="let member of hero.team"> {{member}} </li> </ul>` })Use
@importsinside my.component.css@import 'hero-details-box.css';
ng:view encapsulation Default is ViewEncapsulation.Emulated, rename CSS code to scope the CSS to the component's view. ViewEncapsulation.None adds CSS to the global styles. ViewEncapsulation.Native uses browser's native shadow DOM to attach a shadow DOM to the component's host element.
If the encapsulation is set to ViewEncapsulation.Emulated and the component has no styles nor styleUrls, the encapsulation will automatically be switched to ViewEncapsulation.None.
@Component({ selector: 'toh-heroes', templateUrl: './heroes.component.html', styleUrls: ['./heroes.component.css'], encapsulation: ViewEncapsulation.Emulated }) - Event
(eventType)
https://angular.io/guide/template-syntax#event-binding
(click)="onClick($event)" (input)="name=$event.target.value" (keyup.enter)="addArtist(newArtist.value); newArtist.value=''"
$event is a DOM event object
- Template Reference variable
#var
<input #newArtist (keyup.enter)="addArtist(newArtist.value); newArtist.value=''"> <button (click)="addArtist(newArtist.value); newArtist.value=''">Add</button>
#newArtistcan be used anywhere in the template. - One-way binding
// Instead of <lable>Search <span *ngIf="name">for: {{ name }}</span></label> // Use <lable>Search <span *ngIf="name" [innerHTML]="' for: ' + name"></span></lable> // [innerHTML] can be bind-innerHTML // Or use <span innerHTML="{{ ' for: ' + name }}" // DOM element attribute <img [src]="iconUrl"/> // Bind clicked DOM element to a variable <ul class="heroes"> <li #selectedHero *ngFor="let hero of heroes | async" (click)="onClick(hero, selectedHero)"> <span class="badge">{{hero.id}}</span> {{hero.name}} </li> </ul> // inside export class onClick(item, myElement) { this.name = item.name; myElement.style.backgroundColor="red"; } - Two-way binding
[(x)][(ngModel)][ngModel](ngModelChange)
<input #newArtist [value]="name" (input)="name=$event.target.value" (keyup.enter)="addArtist(newArtist.value); newArtist.value=''"> // It's equivalent to <input #newArtist [(ngModel)]="name" (keyup.enter)="addArtist(newArtist.value); newArtist.value=''">
<input [ngModel]="currentHero.name" (ngModelChange)="setUppercaseName($event)">
ngModel needs ng:FormsModule
- @Input, @Output ng:input output
media-item.component.ts
import { Component, Input } from '@angular/core'; @Component({ selector: 'mw-media-item', templateUrl: './media-item.component.html', styleUrls: ['./media-item.component.css'] }) export class MediaItemComponent { // @Input('mediaItemToWatch') mediaItem; // The mediaItemToWatch is an alias of mediaItem // It's not recommended to have alias @Input() mediaItem; // strict typing @Input() size: number | string; @Output() delete = new EventEmitter(); // Notice delete is defined in sub component and exists inside mw-media-item DOM element onDelete() { // emit an event to the parent app.component.ts this.delete.emit(this.mediaItem); } }app.component.ts
export class AppComponent { onMediaItemDelete(mediaItem) { } firstMediaItem = { id: 1, name: "Firebug", medium: "Series", category: "Science Fiction", year: 2010, watchedOn: 1294166565384, isFavorite: false }; }mediaItem can be used as a new property for binding
[mediaItem]or{{mediaItem}}app.component.html<section> <header> <h1>Media Watch List</h1> <p class="description">Keeping track of the media I want to watch.</p> </header> <mw-media-item [mediaItem]="firstMediaItem" (delete)="onMediaItemDelete($event)"></mw-media-item> </section>media-item.component.html
<h2>{{ mediaItem.name }}</h2> <div>Watched on {{ mediaItem.watchedOn }}</div> <div>{{ mediaItem.category }}</div> <div>{{ mediaItem.year }}</div> <div class="tools"> <a class="delete" (click)="onDelete()"> remove </a> <a class="details"> watch </a> </div> - Structural directives (builtin)
ngIfngForngSwitch<ng-container>
<div *ngIf="mediaItem.watchedOn">Watched on {{ mediaItem.watchedOn }} </div>Even though ngIf is true, only <div> will display
<template [ngIf]="mediaItem.watchedOn"> <div>Watched on {{ mediaItem.watchedOn }} </div> </template>ngFor :: refer to ng:pipe
ngSwitch
<div [ngSwitch]="hero?.emotion"> <happy-hero *ngSwitchCase="'happy'" [hero]="hero"></happy-hero> <sad-hero *ngSwitchCase="'sad'" [hero]="hero"></sad-hero> <confused-hero *ngSwitchCase="'confused'" [hero]="hero"></confused-hero> <unknown-hero *ngSwitchDefault [hero]="hero"></unknown-hero> </div>
Only one structural directive can be applied to one host element
<ng-container>This is almost works like<template><p> I turned the corner <span *ngIf="hero"> and saw {{hero.name}}. I waved </span> and continued on my way. </p>
In
<select>, before we do and dropdown is empty because a browser won't display an<option>within a<span><div> Pick your favorite hero (<label><input type="checkbox" checked (change)="showSad = !showSad">show sad</label>) </div> <select [(ngModel)]="hero"> <span *ngFor="let h of heroes"> <span *ngIf="showSad || h.emotion !== 'sad'"> <option [ngValue]="h">{{h.name}} ({{h.emotion}})</option> </span> </span> </select>
<div> Pick your favorite hero (<label><input type="checkbox" checked (change)="showSad = !showSad">show sad</label>) </div> <select [(ngModel)]="hero"> <ng-container *ngFor="let h of heroes"> <ng-container *ngIf="showSad || h.emotion !== 'sad'"> <option [ngValue]="h">{{h.name}} ({{h.emotion}})</option> </ng-container> </ng-container> </select>
- Attribute directives (builtin)
ngClassngStyle
media-list.component.html
<section> <mw-media-item [ngClass]="{'medium-movies': mediaItem.medium==='Movies', 'medium-series':mediaItem.medium==='Series'}" *ngFor="let mediaItem of mediaItems" [mediaItem]="mediaItem" (delete)="onMediaItemDelete($event)"></mw-media-item> </section>[class]or[style]with a string can also be used<!-- Reset class to badCurly only --> <div class="bad curly special" [class]="badCurly">Bad curly</div> <!-- toggle the "special" class on/off with a property --> <div [class.special]="isSpecial">The class binding is special</div> <button [style.color]="isSpecial ? 'red': 'green'">Red</button> <button [style.background-color]="canSave ? 'cyan': 'grey'" >Save</button> <button [style.font-size.em]="isSpecial ? 3 : 1" >Big</button> <button [style.font-size.%]="!isSpecial ? 150 : 50" >Small</button>
9.14.3.13. Pipe ng:pipe
- Builtin pipes
https://angular.io/api?query=pipe
uppercase, lowercase, titlecase
- Date
date_expression | date[:format]format 'medium': equivalent to 'yMMMdjms' (e.g. Sep 3, 2010, 12:05:08 PM for en-US) 'short': equivalent to 'yMdjm' (e.g. 9/3/2010, 12:05 PM for en-US) 'fullDate': equivalent to 'yMMMMEEEEd' (e.g. Friday, September 3, 2010 for en-US) 'longDate': equivalent to 'yMMMMd' (e.g. September 3, 2010 for en-US) 'mediumDate': equivalent to 'yMMMd' (e.g. Sep 3, 2010 for en-US) 'shortDate': equivalent to 'yMd' (e.g. 9/3/2010 for en-US) 'mediumTime': equivalent to 'jms' (e.g. 12:05:08 PM for en-US) 'shortTime': equivalent to 'jm' (e.g. 12:05 PM for en-US)<p>Today is {{today | date}}</p> <p>Or if you prefer, {{today | date:'fullDate'}}</p> <p>The time is {{today | date:'jmZ'}}</p> - slice
slice:start[:end]
<h2>{{ mediaItem.name | slice:0:10 }}</h2>
- Date
- Custom pipe
https://angular.io/guide/pipes
src/app/search.pipe.ts
import { Pipe, PipeTransform } from '@angular/core'; @Pipe({ name: 'search', // Default is pure, which means the pipe doesn't change the data // pure: true, }) export class SearchPipe implements PipeTransform { // pipe is used in ngFor, the first parameter is the pipe data without specifying in template transform(pipeData, pipeModifier) { return pipeData.filter((eachItem)=> { return eachItem['name'].toLowerCase().includes(pipeModifier.toLowerCase()) || eachItem['reknown'].toLowerCase().includes(pipeModifier.toLowerCase()); }); } // Another example for ngFor pipe. It returns an array transform(mediaItems) { var categories = []; mediaItems.forEach(mediaItem => { if (categories.indexOf(mediaItem.category) <= -1) { categories.push(mediaItem.category); } }); return categories.join(', '); } /* Another transform interface example */ /* transform(value: number, exponent: string): number { let exp = parseFloat(exponent); return Math.pow(value, isNaN(exp) ? 1 : exp); } */ }app.html
<ul class="artistlist cf" *ngIf="query"> <li class="artistlist-item cf" (click)="showArtist(item); query=''" *ngFor="let item of (artists | search: query)"> <artist-item class="content" [artist]=item></artist-item> </li> </ul>
9.14.3.14. Directive ng:directive
- Custom attribute directive
HostListenerHostBinding
It's better to add prefix 'my' to selector name hightlight.directive.ts
import { Directive, ElementRef, HostListener, Input } from '@angular/core'; @Directive({ selector: '[myHighlight]' }) export class HighlightDirective { constructor(private el: ElementRef) { } @Input() defaultColor: string; @Input('myHighlight') highlightColor: string; @HostListener('mouseenter') onMouseEnter() { this.highlight(this.highlightColor || this.defaultColor || 'red'); } @HostListener('mouseleave') onMouseLeave() { this.highlight(null); } private highlight(color: string) { this.el.nativeElement.style.backgroundColor = color; } }app.component.html
<h1>My First Attribute Directive</h1> <h4>Pick a highlight color</h4> <div> <input type="radio" name="colors" (click)="color='lightgreen'">Green <input type="radio" name="colors" (click)="color='yellow'">Yellow <input type="radio" name="colors" (click)="color='cyan'">Cyan </div> <p [myHighlight]="color">Highlight me!</p> <p [myHighlight]="color" defaultColor="violet"> Highlight me too! </p> <hr> <p><i>Mouse over the following lines to see fixed highlights</i></p> <p [myHighlight]="'yellow'">Highlighted in yellow</p> <p myHighlight="orange">Highlighted in orange</p>
Another example using HostBinding which binds host (DOM) property to a getter or another property favorite.directive.ts
import { Directive, HostBinding, Input } from '@angular/core'; @Directive({ selector: '[mwFavorite]' }) export class FavoriteDirective { @HostBinding('class.is-favorite') isFavorite; // Use ~isFavorite = true~ to set a default @Input() set mwFavorite(value) { this.isFavorite = value; } /* @HostBinding('attr.something') get something() { return this.somethingElse; } */ }media-item.component.html
<h2>{{ mediaItem.name }}</h2> <template [ngIf]="mediaItem.watchedOn"> <div>Watched on {{ mediaItem.watchedOn }}</div> </template> <div>{{ mediaItem.category }}</div> <div>{{ mediaItem.year }}</div> <div class="tools"> <svg [mwFavorite]="mediaItem.isFavorite" class="favorite" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"> <path d="M12 9.229c.234-1.12 1.547-6.229 5.382-6.229 2.22 0 4.618 1.551 4.618 5.003 0 3.907-3.627 8.47-10 12.629-6.373-4.159-10-8.722-10-12.629 0-3.484 2.369-5.005 4.577-5.005 3.923 0 5.145 5.126 5.423 6.231zm-12-1.226c0 4.068 3.06 9.481 12 14.997 8.94-5.516 12-10.929 12-14.997 0-7.962-9.648-9.028-12-3.737-2.338-5.262-12-4.27-12 3.737z" /> </svg> <a class="delete" (click)="onDelete()"> remove </a> <a class="details"> watch </a> </div>
9.14.3.15. Model and Custom Component
src/app/app.component.ts
import { Component } from '@angular/core';
import { ArtistItemComponent } from './artists/artist-item/artist-item.component';
// Model
export class Artist {
name: string,
shortname: string,
reknown: string,
bio: string
}
@Component({
selector: 'my-app',
templateUrl: './app.component.html',
styleUrls: ['./app.component.css']
})
export class AppComponent {
artists = ARTISTS;
currentArtist: Artist;
}
var ARTISTS: Artist[] = [...]
src/app/app.component.html
<div class="card search">
<h1 class="search-headline">Artist Directory</h1>
<label class="search-label">search
<span *ngIf="query"
[innerHTML]="' for: ' + query"></span></label>
<input class="search-input"
[(ngModel)]="query" placeholder="type in search term here">
</div><!-- card search -->
<ul class="artistlist cf">
<li class="artistlist-item cf"
*ngFor="let item of artists">
<artist-item class="content" [artist]=item></artist-item>
</li>
</ul>
src/app/app.module.ts
import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { FormsModule } from '@angular/forms';
import { AppComponent } from './app.component';
import { ArtistItemComponent } from './artists/artist-item/artist-item.component';
@NgModule({
imports: [
BrowserModule, FormsModule
],
declarations: [
AppComponent, ArtistItemComponent
],
bootstrap: [
AppComponent
]
})
export class AppModule {}
src/app/artists/artist-item/artist-item.component.ts
import { Component } from '@angular/core';
@Component({
selector: 'artist-item',
templateUrl: './artist-item.component.html',
styleUrls: ['./artist-item.component.css'],
inputs: ['artist']
})
export class ArtistItemComponent {}
src/app/artists/artist-item/artist-item.component.html
<img class="artist-img"
src="images/{{artist.shortname}}_tn.jpg"
alt="{{artist.name}} photo">
<div class="artist-info">
<h2 class="artist-name">{{ artist.name }}</h2>
<h3 class="artist-reknown">{{ artist.reknown }}</h3>
</div>
9.14.3.16. Form
- Template driven
ngModelngSubmit
The form element has a name, then directly use ngModel Template driven form requires ng:FormsModule https://github.com/coursefiles/angular2-essential-training/blob/04_03b/app
https://github.com/coursefiles/angular2-essential-training/blob/04_03b/app/app.module.ts
src/app/media-item-form.component.ts
import { Component } from '@angular/core'; @Component({ selector: 'mw-media-item-form', templateUrl: 'app/media-item-form.component.html', styleUrls: ['app/media-item-form.component.css'] }) export class MediaItemFormComponent { onSubmit(mediaItem) { console.log(mediaItem); } }src/app/media-item-form.component.html
<header> <h2>Add Media to Watch</h2> </header> <form #mediaItemForm="ngForm" (ngSubmit)="onSubmit(mediaItemForm.value)"> <ul> <li> <label for="medium">Medium</label> <select name="medium" id="medium" ngModel> <option value="Movies">Movies</option> <option value="Series">Series</option> </select> </li> <li> <label for="name">Name</label> <input type="text" name="name" id="name" ngModel> </li> <li> <label for="category">Category</label> <select name="category" id="category" ngModel> <option value="Action">Action</option> <option value="Science Fiction">Science Fiction</option> <option value="Comedy">Comedy</option> <option value="Drama">Drama</option> <option value="Horror">Horror</option> <option value="Romance">Romance</option> </select> </li> <li> <label for="year">Year</label> <input type="text" name="year" id="year" maxlength="4" ngModel> </li> </ul> <button type="submit">Save</button> </form> - Model driven, Validation
Model driven form requires ng:ReactiveFormsModule instead of FormsModule. Both can be loaded at the same time
ng:FormGroup ng:FormControl ng:Validators
https://github.com/coursefiles/angular2-essential-training/tree/04_04b/app
https://github.com/coursefiles/angular2-essential-training/blob/04_04b/app/app.module.ts
media-item-form.component.ts
import { Component } from '@angular/core'; import { FormGroup, FormControl, Validators } from '@angular/forms'; @Component({ selector: 'mw-media-item-form', templateUrl: 'app/media-item-form.component.html', styleUrls: ['app/media-item-form.component.css'] }) export class MediaItemFormComponent { form; ngOnInit() { this.form = new FormGroup({ medium: new FormControl('Movies'), name: new FormControl('', Validators.compose([ Validators.required, // this field has to pass Validators.pattern('[\\w\\-\\s\\/]+') ])), // builtin validator /* if it's wrong, DOM will have ng-invalid css class but it's not mandatory name: new FormControl('', Validators.pattern('[\\w\\-\\s\\/]+')), */ category: new FormControl(''), year: new FormControl('', this.yearValidator), }); } yearValidator(control) { if (control.value.trim().length === 0) { return null; } let year = parseInt(control.value); let minYear = 1800; let maxYear = 2500; if (year >= minYear && year <= maxYear) { return null; } else { return { 'year': { min: minYear, max: maxYear } }; } } onSubmit(mediaItem) { console.log(mediaItem); } }media-item-form.component.html
<form [formGroup]="form" (ngSubmit)="onSubmit(form.value)"> <ul> <li> <label for="medium">Medium</label> <select name="medium" id="medium" formControlName="medium"> <option value="Movies">Movies</option> <option value="Series">Series</option> </select> </li> <li> <label for="name">Name</label> <input type="text" name="name" id="name" formControlName="name"> <div *ngIf="form.controls.name.errors?.pattern" class="error"> Name has invalid characters </div> </li> <li> <label for="category">Category</label> <select name="category" id="category" formControlName="category"> <option value="Action">Action</option> <option value="Science Fiction">Science Fiction</option> <option value="Comedy">Comedy</option> <option value="Drama">Drama</option> <option value="Horror">Horror</option> <option value="Romance">Romance</option> </select> </li> <li> <label for="year">Year</label> <input type="text" name="year" id="year" maxlength="4" formControlName="year"> <div *ngIf="form.controls.year.errors?.year" class="error"> Must be between {{form.controls.year.errors?.year.min}} and {{form.controls.year.errors?.year.max}} </div> </li> </ul> <button type="submit" [disabled]="!form.valid">Save</button> </form>
9.14.3.17. Service, Dependency Injection
Either use constructor or @Inject to include a service. Define services in a provider, when service is injected, if it has been created before, reuse it. If not, create it. Service is singleton and once it's been created, it will be injectable across the app's life time.
Unlike componet, directive and pipe, if a service needs other dependencies, the service needs @Injectable decorator. Refer to ng:http:get
Angular services :: Http, FormBuilder, Router
- FormBuilder ng:FormBuilder
- Saves from importing FormControl and FormGroup
import { Component } from '@angular/core'; import { Validators, FormBuilder } from '@angular/forms'; @Component({ selector: 'mw-media-item-form', templateUrl: 'app/media-item-form.component.html', styleUrls: ['app/media-item-form.component.css'] }) export class MediaItemFormComponent { form; constructor(private formBuilder: FormBuilder) {} ngOnInit() { this.form = this.formBuilder.group({ medium: this.formBuilder.control('Movies'), name: this.formBuilder.control('', Validators.compose([ Validators.required, Validators.pattern('[\\w\\-\\s\\/]+') ])), category: this.formBuilder.control(''), year: this.formBuilder.control('', this.yearValidator), }); } yearValidator(control) { if (control.value.trim().length === 0) { return null; } let year = parseInt(control.value); let minYear = 1800; let maxYear = 2500; if (year >= minYear && year <= maxYear) { return null; } else { return { 'year': { min: minYear, max: maxYear } }; } } onSubmit(mediaItem) { console.log(mediaItem); } }
- Custom service and inject variable
https://github.com/coursefiles/angular2-essential-training/tree/05_06b/app https://github.com/coursefiles/angular2-essential-training/tree/05_07b/app
app.module.ts
import { MediaItemService } from './media-item.service'; // inject as a class // inject as a value // https://angular.io/api/core/ValueProvider const lookupLists = { mediums: ['Movies', 'Series'] }; @NgModule({ imports: [ ... ], declarations: [ ... ], providers: [ MediaItemService, { provide: 'lookupListToken', useValue: lookupLists } // Another value provider // , { provide: 'lookupListToken2', useValue: 'abc' } ], bootstrap: [ AppComponent ] }) export class AppModule {}media-item.service.ts
export class MediaItemService { get() { return this.mediaItems; } add(mediaItem) { this.mediaItems.push(mediaItem); } delete(mediaItem) { let index = this.mediaItems.indexOf(mediaItem); if(index >= 0) { this.mediaItems.splice(index, 1); } } mediaItems = [ { id: 1, name: "Firebug", medium: "Series", category: "Science Fiction", year: 2010, watchedOn: 1294166565384, isFavorite: false }, {...} ]; }media-item-list.component.ts
import { Component } from '@angular/core'; import { MediaItemService } from './media-item.service'; @Component({ selector: 'mw-media-item-list', templateUrl: 'app/media-item-list.component.html', styleUrls: ['app/media-item-list.component.css'] }) export class MediaItemListComponent { mediaItems; constructor(private mediaItemService: MediaItemService) {} ngOnInit() { this.mediaItems = this.mediaItemService.get(); } onMediaItemDelete(mediaItem) { this.mediaItemService.delete(mediaItem); } }media-item-form.component.ts
import { Component, Inject } from '@angular/core'; // Inject is for injecting the ValueProvider import { Validators, FormBuilder } from '@angular/forms'; import { MediaItemService } from './media-item.service'; @Component({ selector: 'mw-media-item-form', templateUrl: 'app/media-item-form.component.html', styleUrls: ['app/media-item-form.component.css'] }) export class MediaItemFormComponent { form; constructor( private formBuilder: FormBuilder, private mediaItemService: MediaItemService, @Inject('lookupListToken') public lookupLists) {} // private can be used inside the class here as this.xxx // public can be used outside the class such as in template as xxx ngOnInit() { this.form = this.formBuilder.group({ ... }); } yearValidator(control) { ... } onSubmit(mediaItem) { this.mediaItemService.add(mediaItem); } }media-item-form.component.html
<form [formGroup]="form" (ngSubmit)="onSubmit(form.value)"> <ul> <li> <label for="medium">Medium</label> <select name="medium" id="medium" formControlName="medium"> <option *ngFor="let medium of lookupLists.mediums" value="{{medium}}">{{medium}}</option> </select> </li> ...li </ul> </form> - OpaqueToken, InjectionToken
Use this rather than ValueProvider OpaqueToken is deprecated, use InjectionToken https://angular.io/guide/dependency-injection#injection-token
https://github.com/coursefiles/angular2-essential-training/tree/06_01b/app
providers.ts
import { OpaqueToken } from '@angular/core'; export const lookupListToken = new OpaqueToken('lookupListToken'); export const lookupLists = { mediums: ['Movies', 'Series'] };app.module.ts
import { lookupListToken, lookupLists } from './providers'; @NgModule({ imports: [ ... ], declarations: [ ... ], providers: [ MediaItemService, { provide: lookupListToken, useValue: lookupLists } ], bootstrap: [ AppComponent ] }) export class AppModule {}media-item-form.component.ts
import { lookupListToken } from './provders'; constructor( private formBuilder: FormBuilder, private mediaItemService: MediaItemService, @Inject(lookupListToken) public lookupLists) {} - Http
- mock back end ng:mock back end
https://angular.io/guide/http#in-mem-web-api https://angular.io/api/http/XHRBackend https://github.com/coursefiles/angular2-essential-training/tree/06_03b/app
app.module.ts
import { NgModule } from '@angular/core'; import { BrowserModule } from '@angular/platform-browser'; import { ReactiveFormsModule } from '@angular/forms'; import { HttpModule, XHRBackend } from '@angular/http'; import { AppComponent } from './app.component'; import { MediaItemComponent } from './media-item.component'; import { MediaItemListComponent } from './media-item-list.component'; import { FavoriteDirective } from './favorite.directive'; import { CategoryListPipe } from './category-list.pipe'; import { MediaItemFormComponent } from './media-item-form.component'; import { MediaItemService } from './media-item.service'; import { lookupListToken, lookupLists } from './providers'; // new import { MockXHRBackend } from './mock-xhr-backend'; @NgModule({ imports: [ BrowserModule, ReactiveFormsModule, HttpModule ], declarations: [ AppComponent, MediaItemComponent, MediaItemListComponent, FavoriteDirective, CategoryListPipe, MediaItemFormComponent ], providers: [ MediaItemService, { provide: lookupListToken, useValue: lookupLists }, // new { provide: XHRBackend, useClass: MockXHRBackend } ], bootstrap: [ AppComponent ] }) export class AppModule {}https://github.com/coursefiles/angular2-essential-training/blob/06_03b/app/mock-xhr-backend.ts mock-xhr-backend.ts
- get ng:http:get
http.get() returns an Observable<Response>. Refer to js:observable Observable.map() is one of RxJS operators needed to be imported
RxJS is a 3rd party library implements async Observable pattern.
https://github.com/coursefiles/angular2-essential-training/tree/06_04b/app
It uses the paths defined in ng:mock back end
media-item.service.ts
// new import { Injectable } from '@angular/core'; import { Http } from '@angular/http'; import 'rxjs/add/operator/map'; // This custom service class needs dependency injection, so it needs a decorator Injectable @Injectable() export class MediaItemService { constructor(private http: Http) {} get() { // new return this.http.get('mediaitems') .map(response => { return response.json().mediaItems; }); } add(mediaItem) { this.mediaItems.push(mediaItem); } delete(mediaItem) { let index = this.mediaItems.indexOf(mediaItem); if(index >= 0) { this.mediaItems.splice(index, 1); } } mediaItems = [ {}, ... ]; }media-item-list.component.ts
ngOnInit() { this.getMediaItems(this.medium); } getMediaItems(medium) { this.medium = medium; this.mediaItemService.get() .subscribe(mediaItems => { this.mediaItems = mediaItems; }); } - get with URLSearchParams
https://github.com/coursefiles/angular2-essential-training/tree/06_05b/app
media-item.service.ts
import { Injectable } from '@angular/core'; import { Http, URLSearchParams } from '@angular/http'; import 'rxjs/add/operator/map'; // Other Observable operators // import 'rxjs/add/operator/catch' @Injectable() export class MediaItemService { constructor(private http: Http) {} get(medium) { let searchParams = new URLSearchParams(); searchParams.append('medium', medium); // searchParams.set('medium', medium); return this.http.get('mediaitems', { search: searchParams }) .map(response => { return response.json().mediaItems; }); } add(mediaItem) { ... } delete(mediaItem) { ... } mediaItems = [ {}, ... ]; }media-item-list.component.ts
ngOnInit() { this.getMediaItems(this.medium); } getMediaItems(medium) { this.medium = medium; this.mediaItemService.get(medium) .subscribe(mediaItems => { this.mediaItems = mediaItems; }); } - post, put, delete
https://github.com/coursefiles/angular2-essential-training/tree/07_01b/app
media-item.service.ts
import { Injectable } from '@angular/core'; import { Http, URLSearchParams } from '@angular/http'; import 'rxjs/add/operator/map'; @Injectable() export class MediaItemService { constructor(private http: Http) {} get(medium) { let searchParams = new URLSearchParams(); searchParams.append('medium', medium); return this.http.get('mediaitems', { search: searchParams }) .map(response => { return response.json().mediaItems; }); } add(mediaItem) { // new return this.http.post('mediaitems', mediaItem) .map(response => {}); } delete(mediaItem) { // new return this.http.delete(`mediaitems/${mediaItem.id}`) .map(response => {}); } }media-item-form.component.ts
import { Component, Inject } from '@angular/core'; import { Validators, FormBuilder } from '@angular/forms'; import { MediaItemService } from './media-item.service'; import { lookupListToken } from './providers'; @Component({ selector: 'mw-media-item-form', templateUrl: 'app/media-item-form.component.html', styleUrls: ['app/media-item-form.component.css'] }) export class MediaItemFormComponent { form; constructor( private formBuilder: FormBuilder, private mediaItemService: MediaItemService, @Inject(lookupListToken) public lookupLists) {} ngOnInit() { this.form = this.formBuilder.group({ medium: this.formBuilder.control('Movies'), name: this.formBuilder.control('', Validators.compose([ Validators.required, Validators.pattern('[\\w\\-\\s\\/]+') ])), category: this.formBuilder.control(''), year: this.formBuilder.control('', this.yearValidator), }); } yearValidator(control) { if (control.value.trim().length === 0) { return null; } let year = parseInt(control.value); let minYear = 1800; let maxYear = 2500; if (year >= minYear && year <= maxYear) { return null; } else { return { 'year': { min: minYear, max: maxYear } }; } } onSubmit(mediaItem) { // new this.mediaItemService.add(mediaItem) .subscribe(); } }media-item-list.component.ts
import { Component } from '@angular/core'; import { MediaItemService } from './media-item.service'; @Component({ selector: 'mw-media-item-list', templateUrl: 'app/media-item-list.component.html', styleUrls: ['app/media-item-list.component.css'] }) export class MediaItemListComponent { medium = ''; mediaItems = []; constructor(private mediaItemService: MediaItemService) {} ngOnInit() { this.getMediaItems(this.medium); } onMediaItemDelete(mediaItem) { // new this.mediaItemService.delete(mediaItem) .subscribe(() => { this.getMediaItems(this.medium); }); } getMediaItems(medium) { this.medium = medium; this.mediaItemService.get(medium) .subscribe(mediaItems => { this.mediaItems = mediaItems; }); } }
- mock back end ng:mock back end
9.14.3.18. Route
- base href, config and register routes
https://github.com/coursefiles/angular2-essential-training/tree/07_03b/app
index.html
<base href="/">
app.routing.ts
import { Routes, RouterModule } from '@angular/router'; import { MediaItemFormComponent } from './media-item-form.component'; import { MediaItemListComponent } from './media-item-list.component'; const appRoutes: Routes = [ { path: 'add', component: MediaItemFormComponent }, { path: ':medium', component: MediaItemListComponent }, { path: '', pathMatch: 'full', redirectTo: 'all' } ]; export const routing = RouterModule.forRoot(appRoutes);app.module.ts
import { routing } from './app.routing'; @NgModule({ imports: [ BrowserModule, ReactiveFormsModule, HttpModule, routing ], ... }) - router-outlet
app.component.html Before, both form and list are loaded.
<section> <header> <h1>Media Watch List</h1> <p class="description">Keeping track of the media I want to watch.</p> </header> <mw-media-item-form></mw-media-item-form> <mw-media-item-list></mw-media-item-list> </section>After, form or list is loaded based on the current router state. The loaded component is the next sibling of the router-outlet HTML element.
<section> <header> <h1>Media Watch List</h1> <p class="description">Keeping track of the media I want to watch.</p> </header> <router-outlet></router-outlet> </section> - routerLink
app.component.html
<nav> <a routerLink="/"> <img src="media/04.png" class="icon" /> </a> <a routerLink="/Movies"> <img src="media/03.png" class="icon" /> </a> <a routerLink="/Series"> <img src="media/02.png" class="icon" /> </a> </nav> - read route in component
media-item-list.component.ts
import { Component } from '@angular/core'; import { ActivatedRoute } from '@angular/router'; // new import { MediaItemService } from './media-item.service'; @Component({ selector: 'mw-media-item-list', templateUrl: 'app/media-item-list.component.html', styleUrls: ['app/media-item-list.component.css'] }) export class MediaItemListComponent { medium = ''; mediaItems = []; paramsSubscription; // new constructor( private mediaItemService: MediaItemService, private activatedRoute: ActivatedRoute // new ) {} ngOnInit() { // before // this.getMediaItems(this.medium); // now this.paramsSubscription = this.activatedRoute.params .subscribe(params => { let medium = params['medium']; if(medium.toLowerCase() === 'all') { medium = ''; } this.getMediaItems(medium); }); } // new ngOnDestroy() { this.paramsSubscription.unsubscribe(); } onMediaItemDelete(mediaItem) { this.mediaItemService.delete(mediaItem) .subscribe(() => { this.getMediaItems(this.medium); }); } getMediaItems(medium) { this.medium = medium; this.mediaItemService.get(medium) .subscribe(mediaItems => { this.mediaItems = mediaItems; }); } } - Navigate in code
Form submit and navigate in code media-item-form.component.ts
import { Component, Inject } from '@angular/core'; import { Validators, FormBuilder } from '@angular/forms'; import { Router } from '@angular/router'; // new import { MediaItemService } from './media-item.service'; import { lookupListToken } from './providers'; @Component({ selector: 'mw-media-item-form', templateUrl: 'app/media-item-form.component.html', styleUrls: ['app/media-item-form.component.css'] }) export class MediaItemFormComponent { form; constructor( private formBuilder: FormBuilder, private mediaItemService: MediaItemService, @Inject(lookupListToken) public lookupLists, private router: Router) {} ngOnInit() { this.form = this.formBuilder.group({ medium: this.formBuilder.control('Movies'), name: this.formBuilder.control('', Validators.compose([ Validators.required, Validators.pattern('[\\w\\-\\s\\/]+') ])), category: this.formBuilder.control(''), year: this.formBuilder.control('', this.yearValidator), }); } yearValidator(control) { if (control.value.trim().length === 0) { return null; } let year = parseInt(control.value); let minYear = 1800; let maxYear = 2500; if (year >= minYear && year <= maxYear) { return null; } else { return { 'year': { min: minYear, max: maxYear } }; } } onSubmit(mediaItem) { this.mediaItemService.add(mediaItem) .subscribe(() => { // new this.router.navigate(['/', mediaItem.medium]); }); } }
9.14.3.19. API
9.14.3.20. Style Guide
- File Structure ng:file structure
https://angular.io/guide/styleguide#file-tree
src/app/heroes/hero.component.ts
@Component({ selector: 'toh-hero' }) export class HeroComponent {}src/app/heroes/hero-list/hero-list.component.ts
import { Component, OnInit } from '@angular/core'; import { Hero, HeroService } from '../shared'; @Component({ selector: 'toh-heroes', template: ` <pre>{{heroes | json}}</pre> ` }) export class HeroListComponent implements OnInit { heroes: Hero[] = []; constructor(private heroService: HeroService) { } ngOnInit() { this.heroService.getHeroes().subscribe(heroes => this.heroes = heroes); } }src/app/heroes/shared/hero.service.ts|spec.ts
import { Injectable } from '@angular/core'; import { Http } from '@angular/http'; import { Hero } from './hero.model'; import { ExceptionService, SpinnerService, ToastService } from '../../core'; export class HeroService { constructor(private http: Http) { } getHeroes() { return this.http.get('api/heroes') .map((response: Response) => <Hero[]>response.json().data); } }src/app/heroes/shared/hero.model.ts
export class Hero { id: number; name: string; }src/app/heroes/shared/hero-button.component.ts|html|css|spec.ts
- File name
Use dashes to separate words in file names (hero-list) Use a dot to separate the descriptive name from the type (.component) app/heroes/hero-list.component.ts
A file should have no more than 400 lines of code. A small function should have no more than 75 lines.
- Symbol name and file name
Use upper camel case for class names
// hero-list.component.ts @Component({ ... }) export class HeroListComponent { } // validation.directive.ts @Directive({ ... }) export class ValidationDirective { } // app.module.ts @NgModule({ ... }) export class AppModule // init-caps.pipe.ts @Pipe({ name: 'initCaps' }) export class InitCapsPipe implements PipeTransform {} // user-profile.service.ts @Injectable() export class UserProfileService { } - Bootstrapping
Use main.ts as the file name Include error handling Don't put logic in this file.
import { platformBrowserDynamic } from '@angular/platform-browser-dynamic'; import { AppModule } from './app/app.module'; platformBrowserDynamic().bootstrapModule(AppModule) .then(success => console.log(`Bootstrap success`)) .catch(err => console.error(err)); - Directive selectors (lower camel case)
9.15. React.js
- Documentation
- https://react.dev/
- React Lifecycle methods diagram
- http://projects.wojtekmaj.pl/react-lifecycle-methods-diagram/
- (no term)
- netlify.com, deploy your build folder to a web server. e.g. it has redirect http to https
9.15.1. CDN
9.15.2. create-react-app
- App will be running on local port 3000
- change it in
.envwithPORT=3000
- change it in
- Code structure
- src
- components
- better put each component as a sub directory
- https://www.digitalocean.com/community/tutorials/how-to-create-custom-components-in-react
- MyComponent
- MyComponent.js
- MyComponent.css
- App
- App.js
- App.css
- App.test.js
- (no term)
- index.js
- built code
- src
- Build
react-scriptsis used to build, start, test and eject- Source code is insdie the
create-react-appproject
- Source code is insdie the
# cd to your new project's parent folder cd /path/to/myProjects # new way npx create-react-app lake-app cd lake-app npm start # build a production inside the "build" folder npm run build # then you can serve the prod code. # First install "serve" as a static server sudo npm i -g serve serve -s build
9.15.3. Vite
# uses package create npm create vite@latest my-react-app # or the current directory npm create vite@latest .
9.15.4. react-bootstrap
- https://react-bootstrap.github.io/getting-started/introduction/
- react-boostrap itself does not have .css files
import 'bootstrap/dist/css/bootstrap.min.css';- Or include
<link rel="stylesheet" href="..." />inpublic/index.html
- Or include
In each of your component, import the required BS component
import Jumbotron from 'react-bootstrap/Jumbotron'; // or import { Button, Jumbotron } from 'react-bootstrap';
9.15.5. react-router-dom
- https://reacttraining.com/react-router/web/guides/quick-start
npm i react-router-domIn
src/index.jsorApp.jsimport { BrowserRouter as Router, Switch, Route, Link } from 'react-router-dom';
Inside your component
render () { return ( <Router> <header>some header</header> <Switch> <Route exact path="/" component={Home} /> <Route path="/vitamin" component={Vitamin} /> </Switch> </Router> ) }
9.15.6. JSX
- Comment out
{/*and*/}- (no term)
- Inside
(), use{} - (no term)
- Inside
{}, use()
9.15.6.1. Ternary return with another JSX
useEffect(() => {
count && (document.title = `$(pageTitle) - $(count)`)
})
return (
{this.state.message === 'failed'
? (
<div>
<row>three elements wrapped</row>
<row>inside</row>
<row>another element work.</row>
</div>
)
: (
<row>html like</row>
<row>haiku</row>
<row>must follow rules of structure.</row>
)
}
)
9.15.7. TypeScript
- React TypeScript Cheatsheet
- https://react-typescript-cheatsheet.netlify.app/
9.15.8. CSS
9.15.8.1. Raw CSS file import
// index.js import './components/MyComponent/MyComponent.css'; // Don't use MyComponent.module.css' // components/MyComponent/MyComponent.js // You may import but not necessary because the whole CSS file is imported before in index.js return ( <> <span className="lili">hello</span> </> )
The final HTML is <span class="lili">hello</span>
9.15.8.2. CSS Module Import
- https://github.com/css-modules/css-modules
composes in Sass can be:
@import '/ProfileImage.module.scss' /* your ProfileImage scss file path */ .profileImage { @extend .profileImage; outline: 1px solid red; }
// components/MyComponent/MyComponent.js import styles from './MyComponent.module.css'; return ( <> <span className={[styles.lili, "lilihello"].join(' ')}>hello</span> </> )
The final HTML is <span class="MyComponent_lili__J4Vn2 lilihello">hello</span>
The naming convention is [fileName]_[cssClassName]__[hash]
9.15.8.3. Styled-Component (CSS in JS)
- https://www.smashingmagazine.com/2020/07/styled-components-react/
- Styled-components is a CSS-in-JS tool
9.15.8.4. NPM classnames
- https://www.npmjs.com/package/classnames#usage-with-reactjs
Compare
class Button extends React.Component { // ... render () { var btnClass = 'btn'; if (this.state.isPressed) btnClass += ' btn-pressed'; else if (this.state.isHovered) btnClass += ' btn-over'; return <button className={btnClass}>{this.props.label}</button>; } } // Using NPM classnames var classNames = require('classnames'); class Button extends React.Component { // ... render () { var btnClass = classNames({ btn: true, 'btn-pressed': this.state.isPressed, 'btn-over': !this.state.isPressed && this.state.isHovered }); return <button className={btnClass}>{this.props.label}</button>; } }
9.15.8.5. NPM styled-components
- Install
npm i styled-components
- GitHub
- https://styled-components.com/
- https://styled-components.com/docs
9.15.9. Redux
- Redux Toolkit
- Redux core
- install it to the current project
- UMD build (CDN version) where Redux Toolkit is available as
window.RTKon browsers
- UMD build (CDN version) where Redux Toolkit is available as
- react-redux
- A complimentary package which deals with binding
npm i -S react-redux
9.15.10. Custom hooks
9.15.10.1. useState
- Group states and state setters in a file
- usePasswordToggler
// ./src/hooks/usePasswordToggler.js import {useState} from 'react'; export const usePasswordToggler = () => { const [passwordVisibility, setPasswordVisibility] = useState(true); const [type, setType] = useState('password'); const handlePasswordVisibility = () => { if (type === 'password') { setType('text'); setPasswordVisibility(!passwordVisibility); } else if (type === 'text') { setType('password'); setPasswordVisibility(!passwordVisibility); } }; return { type, passwordVisibility, handlePasswordVisibility }; };
import React from 'react'; import {usePasswordToggler} from './hooks/usePasswordToggler'; import './App.css'; function App() { const {type, passwordVisibility, handlePasswordVisibility} = usePasswordToggler() return ( <main> <div> <input type={type} placeholder='Enter password...' /> <button onClick={handlePasswordVisibility}>{passwordVisibility ? 'Show' : 'Hide'} Password</button> </div> </main> ); }
9.15.10.2. useReducer
- Consolidate all state update logic outside the component in a single function, called a reducer
- https://react.dev/learn/extracting-state-logic-into-a-reducer
- Reducer function catches the fired event and change the state
- A reducer must be a pure function
- Sample
Old code
import { useState } from 'react'; import AddTask from './AddTask.js'; import TaskList from './TaskList.js'; let nextId = 3; const initialTasks = [ {id: 0, text: 'Visit Kafka Museum', done: true}, {id: 1, text: 'Watch a puppet show', done: false}, {id: 2, text: 'Lennon Wall pic', done: false}, ]; export default function TaskApp() { const [tasks, setTasks] = useState(initialTasks); function handleAddTask(text) { setTasks([ ...tasks, { id: nextId++, text: text, done: false, }, ]); } function handleChangeTask(task) { } function handleDeleteTask(taskId) { } return ( <> <AddTask onAddTask={handleAddTask} /> <TaskList tasks={tasks} onChangeTask={handleChangeTask} onDeleteTask={handleDeleteTask} /> </> ) }
- Step 1
Move from setting state to dispatching actions
function handleAddTask(text) { dispatch({ type: 'added', id: nextId++, text: text, }); }
- Step 2
Write a reducer function which groups all dispatching events that change the state
function tasksReducer(tasks, action) { switch (action.type) { case 'added': { return [ ...tasks, { id: action.id, text: action.text, done: false, }, ]; } case 'changed': { return ''; } default: { throw Error('Unknown action: ' + action.type); } } }
- (no term)
Step 3: Use the reducer
import { useReducer } from 'react'; import AddTask from './AddTask.js'; import TaskList from './TaskList.js'; let nextId = 3; const initialTasks = [ {id: 0, text: 'Visit Kafka Museum', done: true}, {id: 1, text: 'Watch a puppet show', done: false}, {id: 2, text: 'Lennon Wall pic', done: false}, ]; export default function TaskApp() { const [tasks, dispatch] = useReducer(tasksReducer, initialTasks); function handleAddTask(text) { dispatch({ type: 'added', id: nextId++, text: text, }); } function handleChangeTask(task) {} function handleDeleteTask(taskId) {} return ( <> <h1>Prague itinerary</h1> <AddTask onAddTask={handleAddTask} /> <TaskList tasks={tasks} onChangeTask={handleChangeTask} onDeleteTask={handleDeleteTask} /> </> ); }
9.15.10.3. useContext
- Use context to pass properties from parent component to distant children
- https://react.dev/learn/passing-data-deeply-with-context
- LevelContext.js
createContextto initiate context
<LevelContext.Provider value={initialLevel}>- In parent component
- If any component inside this parent component asks for LevelContext, give them this
initialLevel. The component will use the value of the nearest<LevelContext.Provider>in the UI tree above it.
9.15.10.4. useContext and useReducer
9.15.11. Use absolute imports
By default, React apps with nested directory structures might lead to confusion on the import path such as the example below:
import { Button } from '../../components';
Specify base path in jsconfig.json:
{
"compilerOptions": {
"baseUrl": "src"
},
"include": ["src"]
}
import { Button } from 'components';
React picks up jsconfig.json or tsconfig.json to resolve path in config/paths.js
If your project includes webpack.config.js at the root, you may further personalize it by using a prefix such as @components or ~components. See the configuration below:
module.exports = {
resolve: {
extensions: ['js'],
alias: {
'@': path.resolve(__dirname, 'src'),
'@components': path.resolve(__dirname, 'src/components')
'@hooks': path.resolve(__dirname, 'src/hooks'),
}
}
};
import { Button } from '@components';
9.15.12. Refactor
9.15.13. Incrementally Migrate to React
- Opptunities of migration
- Faster browser rendering speed (open loan > close loan > reopen loan)
- PHP returns data only instead of bytes of HTML
- React only renders the parts of HTML that are changed
- e.g. a list of borrowers under loan view, remove a borrower from loan, PHP returns the whole Borrower section again. Instead, PHP returns the updated data of the loan's borrowers, and React will remove that removed Borrower div. Less amount of data is transferred, less re-rendering
- Front end and Back end decoupling
- Reduce risk
- Remove a lot of spaghetti PHP code that was mixed with HTML
- Provide an opportunity for us to rewrite the PHP code and standardize the data
- Ship new frontend features faster
- Reduce risk
- Faster browser rendering speed (open loan > close loan > reopen loan)
- Risk of migration to React
- Risk in rewriting PHP code to output/supply data
- One challenge is the action rewrite, e.g. JavaScript and PHP delloanclient(clientId, loanId) returns HTML of borrowers of a loan, now we need to return data and then update the state.
- A/B testing, Laravel Facade concept, Adaptor Pattern, delloanclientv1, delloav2, delloanclient
- Risk in rewriting PHP code to output/supply data
- Metrics
- Loan view time to interactive
- Size of the app
- memory usage
- state memory
- number of event handlers
- Step by Step
- Choose small basic components first
- Basic UI components like buttons
- Lowest hanging fruits
- New feature has to have front end and back end decoupled
- What's the data we should provide?
- New feature needs basic UI components. Develop those.
- Choose small basic components first
- Data
- Actions of Data
- CRUD
- Form
- In other words, data first
- Actions of Data
9.15.14. React State data structure
- gss
- country
- almodules
- loans
- loans table row in almost full detail (hide not-safe-for-public fields)
- loanid: 123
- borrowers
- not the full borrower detail
- id:
- order:
- loanid: 123
- loans table row in almost full detail (hide not-safe-for-public fields)
- borrowers
- borrower in full detail
- client_id
- 756
- first_name
- last_name
- corporation_name
- borrower in full detail
9.15.15. Directory Structure
- src
- assets
- components
- Input
- a component called Input
- Input.js
- all the logic
- (no term)
- Input.css
- (no term)
- Input.test.js
- pages
- Strictly views, no business logic
- They are also components
- codes that interact with external API resources
- utils
- hooks
- store
9.15.16. Tools
- Chrome Extension: React Developer Tools
- https:://react.new
- Enable Chrome > More Tools > Rendering > Paint flashing
- https://openreplay.com/
- Redux-saga to manage a large amount of asynchronous code and side effects
- https://www.mindbowser.com/redux-saga-vs-redux-thunk/
- https://react-icons.github.io/react-icons/
9.16. CryptoJS NPM
9.17. CryptoJS Goolge Code Archive
9.18. Moment.js
10. NodeJS
10.1. Install
10.1.1. Ubuntu
apt-get update # 18.04 has the first installed but not the libssl-dev apt-get install build-essential libssl-dev # Just go to https://github.com/nvm-sh/nvm to run the install script curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash source ~/.bashrc
- Directory
~/.nvmis created with the cloned code - Insert something to the corresponding profile file
~/.bashrc or bash_profile or .profile or .zshrc
- Log out and back again or source the file so that current session knows changes
- List Node.js available versions
nvm ls-remoteand look for version with Latest LTS
# nvm install 10.14.1 # nvm use 10.14.1 nvm install v16.14.0 # What versions are installed nvm ls # Current in-use node version node -v # Make one version as default for new sessions nvm alias default 6.10.2 # Use the default version nvm use default # to uninstall, first make sure the target version is not the current active version nvm current nvm uninstall node_version # if the target version is the current active version, first deactivate nvm deactivate nvm uninstall node_version
Global packages installed using npm install -g express are in ~/.nvm/versions/node/node_version/lib/node_modules/express
10.1.1.1. Install using PPA
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - sudo apt-get install -y nodejs # you might need to install gnupg sudo apt-get install -y gnupg # might need to install npm if nodejs doesn't have it sudo apt-get install -y npm nodejs -v npm -v # install build-essential so that npm can compile code from source sudo apt install build-essential
10.1.2. MacOS
brew update # other alias: nodejs # will install the latest version (not the LTS version) brew install node # node, npm and npx are installed node -v npm -v npx -v
10.1.3. Install on Windows
Download the .msi and install with NPM. Installation path: C:\Program Files\nodejs\
Download the latest .msi to upgrade. And upgrade any existing global packages.
Upgrade npm on Windows
- Just use the latest .msi to upgrade is good enough to upgrade npm for Windows
- https://www.npmjs.com/package/npm-windows-upgrade
- PowerShell with admin
Set-ExecutionPolicy Unrestricted -Scope CurrentUser -Force npm install --global --production npm-windows-upgrade
Refer to phpstorm:nodejs
10.2. Basic
- Modules installed globally usually have an executable command for direct use.
- For locally installed modules, usually run
./node_modules/module_name/bin/exec_cmd
- For locally installed modules, usually run
- https://node.green/
- Find V8 versio
node -p process.versions.v8showsV8.ver.number-node.nodeVersionNumber- More
- https://nodejs.org/en/docs/es6/
node -v npm -v # clear cache npm cache clean --force # or npm cache verify # Install npm package to ./node_modules npm install express # install without arguments will install dependencies including devDependencies npm install # with flag or when NODE_ENV is set to production, it will not install devDependencies npm install --production # with --only={prod[uction]|dev[elopment]} to install only non-devDependencies or devDependencies npm install --only=dev # Install package globally (available to other projects using the same Node.js version) # ~/.nvm/node_version/lib/node_modules/package_name npm install -g gulp # What packages/modules are installed globally, just remove -g for locally npm ls -g # Check which global packages are outdated npm outdated -g # Update all global packages. Some global package update requires removing first npm update -g # Update one global package npm install -g <packagename> # Remove a package globally, just remove -g for locally npm remove async -g
Link
# run npm link alone in local repo is to create a symlink in global folder that links to the package where the command was executed cd ~/projects/node-redis # Note local package name is not node-redis but redis as in ~/projects/node-redis/package.json npm link # create symlink in global folder so that redis can be globally referrenced/linked. cd ~/projects/node-bloggy # go to another project npm link redis # install redis package in ~/projects/node-bloggy/node_modules/node-redis, file changes in ~/projects/node-redis/ will be reflected. npm link express # a global package can be linked inside any local repo
10.3. Global Object
console.log is actually global.console.log https://nodejs.org/api/globals.html
10.3.1. __dirname, __filename
Full path of the directory/file of the current module/file
10.3.2. console nodejs:console
Backtick
var a = "World"; console.log(`Hello ${a}`); // Hello World // console.dir(obj[, options]) uses util.inspect() console.dir(a, { showHidden: true, depth: 2, // number, can be null colors: false });
10.3.3. require
const myImport = require('path/to/file')- Along with module.exports is CommonJS (CJS) module system. See CommonJS
- Other module systems
- See JavaScript
import
10.3.3.1. CommonJS
- ES5 (old)
- https://blog.risingstack.com/node-js-at-scale-module-system-commonjs-require/
- Features
- modules are imported synchronously
requiregives a copy of the imported object- CJS will not work in the browser. It's for Server side apps.
// Importing a local module with a path relative to the `__dirname` or current // working directory. (On Windows, this would resolve to .\path\myLocalModule.) const myLocalModule = require('./path/myLocalModule'); // Importing a JSON file: const jsonData = require('./path/filename.json'); // Importing a module from node_modules or Node.js built-in module: const crypto = require('crypto');
10.3.3.2. Asynchronous Module Definition (AMD)
- Features
- Asynchronous and made for frontend. Exact opposite of CJS
Code example
// define is a RequireJS function // Define one module define(['dependency1', 'dependency2'], function(dependency1, dependency2) { // Methods function myPublicMethod() { console.log('Hello, world!'); } // Exposed public methods return { myPublicMethod: myPublicMethod }; });
10.3.3.3. Universal Module Definition (UMD)
- Features
- For apps that work on both synchronous and asynchronous environment. Browser and Server apps.
- It's more like a pattern to configure several module systems
Code example
(function (root, factory) { if (typeof define === 'function' && define.amd) { // AMD. Register as an anonymous module. define(['dependency'], factory); } else if (typeof module === 'object' && module.exports) { // Node. Does not work with strict CommonJS, but // only CommonJS-like environments that support module.exports, // like Node. module.exports = factory(require('dependency')); } else { // Browser globals (root is window) root.returnExports = factory(root.dependency); } }(this, function (dependency) { // Methods function myModule() { console.log('Hello, world!'); } // Exposed public methods return { myModule: myModule }; }));
10.3.3.4. ECMAScript Module (ESM)
- ES6 (new)
- Features
- Part of ECMAScript specification and is supported in modern JavaScript environments e.g. Node.js and modern web browsers.
exportandimportare used to define and import modules - Asynchronous
- Part of ECMAScript specification and is supported in modern JavaScript environments e.g. Node.js and modern web browsers.
- Code example
// https://stackoverflow.com/questions/39282253/how-can-i-alias-a-default-import-in-javascript import {default as myAlias} from 'my-module'; // eq. to import myAlias from 'my-module'; // React is the react.default and useEffect is one of the exports import React, {useEffect} from 'react'; // e.g. MyReact.ReactElement, but not MyReact.default import * as MyReact from 'react'; // a more complicated example // material/Chip/index.js export { default } from './Chip'; // Chip.js, default is const Chip. Then index.js exported default is const Chip export { default as chipClasses } from './chipClasses'; export * from './chipClasses'; // export default using arrow function export default () => { } // eq. export default function MyComponent() { } export function MyComponent() { } // eq. (I don't like it) export const MyComponent = () => { } // MyComponent/MyComponent.js import Chip from '../material/Chip';
Import from ES5
// tailwindcss/forms module.exports = forms import whateverNameYouWant from '@tailwindcss/forms'
10.3.4. process nodejs:process
console.log(process.pid)process.versions.node
10.3.4.1. process.argv
node app.js --user li --greeting "Good Day Sir"
app.js
console.log(process.argv); // ['nodejs installation full path', 'full path to module', '--user', 'li', '--greeting', 'Good Day Sir'] const grab = flag => { let index = process.argv.indexOf(flag)+1; return process.argv[index]; } var greeting = grab('--greeting');
10.3.4.2. process.stdout, process.stdin, process.exit(), process.on()
var q = ['Q1', 'Q2']; var a = []; function ask(i) { process.stdout.write(`\n\n ${q[i]}`); process.stdout.write(' > '); } process.stdin.on('data', function(data) { answers.push(data.toString().trim()); if (a.length < q.length) { ask(a.length); } else { process.exit(); } }); process.on('exit', function() { process.stdout.write(`${a[0]}, ${a[1]}, ${a[2]}`); }); ask(0);
10.3.4.3. process.stdout.clearLine(), process.stdout.cursorTo(), process.stdout.write()
var currentTime = 0; var percentWaited = 0; function writePercent(p) { process.stdout.clearLine(); process.stdout.cursorTo(0); process.stdout.write(`waiting ... ${p}%`); } var interval = setInterval(functino() { currentTime += 500; percentWaited = Math.floor((currentTime/3000) * 100); writePercent(percentWaited); }, 500); setTimeout(function() { clearInterval(interval); writePercent(100); console.log("\n\n done \n\n"); }, 3000); writePercent(percentWaited);
10.3.5. module
./lib/Person.js
// some code module.exports = Person; // literal object or any Javascript type // even export an anonymous function will work // module.exports is the object that is returned by the require statement
./app.js
var Person = require('./lib/Person'); // do not include .js var ben = new Person('Ben'); var george = new Person('George');
10.4. Modules, Packages, Frameworks
10.4.1. Get version
npm list- locally installed packages
npm list -g- globally installed packages
npm list -g | grep gulp- to see if a package is installed globally
npm list -g --depth=0- global installed packages at first level
- (no term)
npm list expressnpm ls -depth 0- including devDependencies
npm ls -dev- devDependencies only
npm ls -dev -depth 0- list first level devDependencies
npm view PKG_NAME dependencies- list dependencies of a package
npm view PKG_NAME versions --json- The latest available version of a package
- view aliaes
- info, show, v
- Distribution tags
npm view mjml dist-tags- latest
- no pre-release/beta versions
- beta
- optional
- future
- optional
- next
- usually is set to a beta version to be the next version
- (no term)
- Backup and remove the node_modules and run
npm install mjml@next - Update npm
sudo npm i -g npm- (no term)
- Install packages
npm i -D- eq. to
npm install --save-dev gulp-imagemin- devDependencies are the ones only needed for development not in production or testing environments
- (no term)
- By default, the dependency is saved since v5~npm i -S~
- Legacy
npm i -S gulp-imagemineq.npm install --save gulp-imagemin
- Npm website for a package
gulp-sourcemaps - https://www.npmjs.com/package/gulp-sourcemaps
10.4.2. Semver ~ vs ^
^1.2.3- any 1.x.x latest release but will miss 2.0.0 (e.g. allow minor and patch but not major)
~1.2.3- any 1.2.x versions but will miss 1.3.0 (e.g. allow patch but not minor or major)
^- caret, is the default now
10.4.3. package.json
10.4.4. Update Package
npm outdated -l- Show if there're updates for locally installed packages based on package.json in
--longformat- package status
- Current
- it may be higher than Latest as it's next in dist-tags above
- same as latest in dist-tags above
- A package is outdated when its Current is lower than Latest. It'll update to Latest. If Current is higher than Latest, then it'll be downgraded. package.json also limits what versions can be updated to so don't be surprised after
npm updatethere're still records innpm outdated
- package status
npm outdated --global- for globally installed packages
npm outdated -g --depth=0- see which top level globally installed packages need to be updated
npm update -g <package_name>- update a single global package
npm update -g- updaste all global packages
- Some packages require administrative privilege. On Windows open PowerShell as Admin
npm update- update all outdated "dependencies" in package.json and change package.json to save the new version as the minimum required dependency.
npm update --dev- updates all outdated "dependencies" and "devDependencies" in package.json
npm update sharp- update a local package to the latest
- (no term)
- General package update process
npm outdated- see which packages have updates.
npm outdated -lfor long format- compare what is installed with package.json
npm update- update packages and package.json
npm audit- check vulnerabilities
npm install- sometimes it's needed..
- (no term)
- Resolve vulnerabilities after
npm audit fixYou should rely on first-level contributers to solve the problem. Make sure you update the first-level packages. If you still want to resolve it, cautiously follow these steps:
npm auditfind the problematic packages from the last part ofPath, e.g. packageminimistnpm ls minimistnpm i minimist --save-dev- Add this to package.json
"resolutions": { "minimist": "^1.2.5" }
- npm users need to install this package
npm i npm-force-resolutions --save-dev, add this to package.json
"scripts": { "preinstall": "npx npm-force-resolutions" }
- Try delete node_modules and package-lock.json
npm install,npm update,npm audit fixandnpm audit- Some vulnerabilties cannot be solved
- First-level package has hard coded the dependency version number
- No patch available yet
10.4.5. Scoped package
npm install -g @angular/cli where @ indicates angular/cli is a scoped package.
Without @, package is installed in node_modules/packagename
Scoped package is installed in node_modules/@angular/cli
Thus, scoped package can group packages under one namespace
package.json
"dependencies": { "@angular/cli": "^1.3.0" }
In code
require('@angular/cli')
10.4.6. Peer dependencies
A dependency says "I need this thing directly available to me" A peerDependency says "If you want to use me, you need this ting available to you"
However, as of verison 3, npm does not install packages listed in peerDependencies sections.
But npm issues a warning when
- When any peer dependencies are missing
- or when the application or any of its other dependencies installs a different version of a peer dependency.
It is your job to copy all peer dependency packages to devDependencies to install them.
For example, you app depends on angular and your app doesn't have peerDependencies. However, angular has peerDependencies.
You will make sure to list any angular's peerDependencies into your app's devDepenedencies
10.4.7. Core modules
10.4.7.1. v8 - core nodejs:v8
var v8 = require('v8'); console.log('v8.getHeapStatistics()');
10.4.7.2. util - core nodejs:util
var EventEmitter = require('events').EventEmitter; const util = require('util'); var Person = function(name) { this.name = name; }; util.inherits(Person, EventEmitter); // Add EventEmitter to Person's prototype // Person inherits all from EventEmitter var ben = new Person("Ben"); ben.on('speak', function(said) { console.log(`${this.name} said ${said}`); }); ben.emit('speak', "I'm Ben"); // fully print an object console.log(util.inspect(myObj, {showHidden: false, depth: null})); // or this for short cosole.log(util.inspect(myObj, false, null)); // console.dir actually uses util.inspect. See nodejs:console util.format('%s:%s', 'foo'); // 'foo:%s' util.format('%s:%s', 'foo', 'bar', 'baz'); // extra arguments are coerced into strings delimited by a space // 'foo:bar baz' // %s > string, %d > integer or floating point value // %i > integer, %f > floating point value // %j > JSON // %% > percent sign '%'
10.4.7.3. path - core nodejs:path
var path = require('path'); // path is native, get just the file name console.log(path.basename(__filename)); // No trailing '/' var dirUploads = path.join(__dirname, 'www', 'files', 'uploads');
10.4.7.4. readline - core nodejs:readline
rl.createInterface(options)rl.question(query, callback)- Events
- close
var readline = require('readline'); var rl = readline.createInterface(process.stdin, process.stdout); var aPerson = { name: '', sayings: [] } rl.question("What is the name of a real person? ", function(a) { aPerson.name = a; rl.setPrompt(`What would ${aPerson.name} say? `); rl.prompt(); // display prompt rl.on('line', function(saying) { aPerson.sayings.push(saying.trim()); if (saying.toLowerCase().trim() === 'exit') { rl.close(); } else { rl.setPrompt(`What else would ${aPerson.name} say? ('exit' to leave) `); rl.prompt(); } }); }); rl.on('close', function() { console.log("5s is a real person that says %j", aPerson.name, aPerson.sayings); process.exit(); });
10.4.7.5. events - core nodejs:events
e.EventEmitter()- emitter class
- emitter.on('eventName', callback)
- emitter.emit('eventName', var1, var2)
var events = require('events'); var emitter = new events.EventEmitter(); emitter.on('customEvent', (msg, status) => { console.log(`${status}: ${msg}`); }); emitter.emit('customEvent', "Hello", 200);
10.4.7.6. fs - core nodejs:fs
- Default fs is not a Promise although it is async with callback
fs.readdir( './abc', callback)- They cannot be chained. Consider nodejs:bluebird or nodejs:co-fs
- Every function has a sync version. Just append
Syncat the endlet theFiel = fs.readdirSync('./abc')
- fs.readFile('path', 'utf-8', callback(err, data))
- fs.readdir('path', callback(err, data))
- stats
- stats.isFile()
- fs.writeFile('path', data)
- fs.rename
- fs.unlink
- fs.appendFile
- fs.mkdir, fs.rmdir
- fs.createReadStream, fs.createWriteStream
var fs = require('fs'); var data = require('./data1.json'); console.log(data.name); // data is an object fs.readFile('./data1.json', 'utf-8', (err, data) => { // without specifying utf-8, file will be read as binary data = JSON.parse(data); // data is string console.log(data.name); // res.setHeader('Content-Type', 'text/html'); // res.send(html); }); // Read directory fs.readdir('/root', function(err, data) { console.log(data); // data is array data.forEach(function(fname) { var file = path.join(__dirname, "lib", fname); var stats = fs.statSync(file); if (stats.isFile() && fname !== '.DS_Store') { fs.readFile(file, "UTF-8", function(err, data) { console.log(data); }); } }); }); // Write file var tomString = '{ "name": "Tom" }'; fs.writeFile('tom.json', tomString, function(err) {}); var timmy = { name: 'Timmy' }; fs.writeFile('timmy.json', JSON.stringify(timmy)); // Append file fs.appendFile('timmy.json', 'more text \n'); // Rename file fs.rename('./lib/project-config.js', './lib/config.json'); // Move file fs.rename('./lib/project-config.js', './config.json'); // Remove file fs.unlink('./lib/config.json'); // New folder if (!fs.existsSync("lib")) { fs.mkdir("lib", function(err) {}); } // Rename or move fs.rename('./assets/logs', './logs'); // Remove folder, folder must be empty fs.rmdir('./logs'); // Remove all files from a directory fs.readdirSync('./logs').forEach(function(fileName) { fs.unlinkSync('./logs/' + fileName); }); // File read stream var stream = fs.createReadStream('./chat.log', 'UTF-8'); var data = ""; stream.once('data', function() { console.log('Start reading file'); }); stream.on("data", function(chunk) { process.stdout.write(` chunk: ${chunk.length} |`); data += chunk; }); stream.on('end', function() { console.log(data.length); }); // File write stream var stream = fs.createWriteStream('result.md'); stream.write(`${data}\n`); stream.close();
10.4.7.7. child_process - core nodejs:child_process
Exec is for short process
var exec = require('child_process').exec; exec('ls', function(err, stdout) { if (err) throw err; console.log(stdout); });
Spawn for ongoing process (async)
var spawn = require('child_process').spawn; var cp = spawn('node', ["app"]); // node app cp.stdout.on("data", function(data) { // Output child process stdout console.log(`STDOUT: ${data.toString()}`); }); cp.on("close", function() { console.log("Child process has ended."); process.exit(); }); setTimeout(function() { // maximum run time cp.stdin.write("stop"); }, 4000);
10.4.8. npx
- Run
<command>either from a local./node_modules/.bin/<command>, or from a central cache, installing any packages needed in order for<command>to run - By default, it will check whether
<command>exists in$PATH, or in the local project binaries, and execute that. If<command>is not found, it will be installed prior to execution
# should already be installed npm i -g npx # usage # npx [options] <command>[@version] [command-arg]... # npx [options] [-p|--package <pkg>]... <command> [command-arg]... # npx [options] -c '<command-string>' # npx --shell-auto-fallback [shell]
10.4.9. BabelJS
- Try out
- https://babeljs.io/
- CDN
- https://babeljs.io/docs/babel-standalone
- (no term)
- Babel vs TypeScript
- Babel doesn't check for types
- Babel only compiles one file at a time while TS compiles an entire project
- (no term)
- Modules
npm i -D @babel/coreconst babel = require("@babel/core"); babel.transform("code", optionsObject);
const babel = require("")
- allows to run CLI
- e.g.
./node_modules/.bin/babel src --out-dir libtransform files from src and output to lib folder
- e.g.
- https://babeljs.io/docs/en/next/babel-node.html
- Like CLI, but it enables to call Babel inside Node e.g.
nodemon --exec babel-node index.js
- Like CLI, but it enables to call Babel inside Node e.g.
- Plugins & Presets
npm i -D @babel/preset-env- https://babeljs.io/docs/en/babel-preset-env
- Then
./node_modules/.bin/babel src --out-dir lib --presets=@babel/env- Other presets
- https://www.npmjs.com/search?q=babel-preset
10.4.9.1. Config file
- Project-wide config
babel.config.json- useful when there're multiple package directories e.g. multiple
package.json
- File-relative config
.babelrc.json.babelrcis an alias- (no term)
package.jsonfile with ababelkey
{
"presets": [
"@babel/preset-env"
]
}
10.4.10. axios
npm i axiosimort axios from 'axios'axios.post(url[, data[, config]])- Request config
10.4.11. lodash nodejs:lodash
# npm i -g npm npm i --save lodash
var _ = require('lodash'); var users = [ { 'user': 'barney', 'age': 36, 'active': true }, { 'user': 'fred', 'age': 40, 'active': false }, { 'user': 'pebbles', 'age': 1, 'active': true } ]; _.find(users, function(o) { return o.age < 40; }); // => object for 'barney' // The `_.matches` shorthand. _.find(users, { 'age': 1, 'active': true }); // => object for 'pebbles' // The `_.matchesProperty` shorthand. _.find(users, ['active', false]); // => object for 'fred' // The `_.property` shorthand. _.find(users, 'active'); // => object for 'barney' var user = _.findIndex(users, {id: req.params.id}); var update = req.body; if (update.id) { delete update.id; } if (!users[user]) { res.send('user not found'); } else { var updatedUser = _.assign(users[user], update); // merge multiple objects from left to right for enumerable string keyed properties res.json(updatedUser); }
10.4.11.1. _.identity(value)
returns the first argument it receives
var object = { 'a': 1 };
console.log(_.identity(object) = object); // => true
10.4.11.2. _.matches(source)
creates a function that performs a partical deep comparison between a given object and source, return true if the given object has equivalent prop values, else false. Partial comparisons will match empty array and empty object source values against any array or object value, respectively.
var objects = [ { 'a': 1, 'b': 2, 'c': 3 }, { 'a': 4, 'b': 5, 'c': 6 } ]; _.filter(objects, _.matches({ 'a': 4, 'c': 6 })); // => [{ 'a': 4, 'b': 5, 'c': 6 }]
10.4.11.3. _.find, _.findLast
Return the first element that the predicate function returns true.
_.find(collection, [predicate=_.identity], [fromIndex=0]) _.findLast(collection, [predicate=_.identity], [fromIndex=collection.length-1]) // Predicate function is passed with (value, index|key, collection) var users = [ { 'user': 'barney', 'age': 36, 'active': true }, { 'user': 'fred', 'age': 40, 'active': false }, { 'user': 'pebbles', 'age': 1, 'active': true } ]; _.find(users, function(o) { return o.age < 40; }); // => object for 'barney' // The `_.matches` iteratee shorthand. _.find(users, { 'age': 1, 'active': true }); // => object for 'pebbles' // The `_.matchesProperty` iteratee shorthand. _.find(users, ['active', false]); // => object for 'fred' // The `_.property` iteratee shorthand. _.find(users, 'active'); // => object for 'barney'
10.4.11.4. _.iteratee
Creates a function that invokes func with the arguments of the created function. func is
- a property name
- the created function returns the prop value for a given element.
_.iteratee([func=_.identity]) var users = [ { 'user': 'barney', 'age': 36, 'active': true }, { 'user': 'fred', 'age': 40, 'active': false } ]; // The `_.matches` iteratee shorthand. _.filter(users, _.iteratee({ 'user': 'barney', 'active': true })); // => [{ 'user': 'barney', 'age': 36, 'active': true }] // The `_.matchesProperty` iteratee shorthand. _.filter(users, _.iteratee(['user', 'fred'])); // => [{ 'user': 'fred', 'age': 40 }] // The `_.property` iteratee shorthand. _.map(users, _.iteratee('user')); // => ['barney', 'fred'] // Create custom iteratee shorthands. _.iteratee = _.wrap(_.iteratee, function(iteratee, func) { return !_.isRegExp(func) ? iteratee(func) : function(string) { return func.test(string); }; }); _.filter(['abc', 'def'], /ef/); // => ['def']
10.4.11.5. _.property(path)
Creates a function that returns the value at path of a given object
var objects = [ { 'a': { 'b': 2 } }, { 'a': { 'b': 1 } } ]; _.map(objects, _.property('a.b')); // => [2, 1] _.map(_.sortBy(objects, _.property(['a', 'b'])), 'a.b'); // => [1, 2]
10.4.12. lighthouse npm:lighthouse
https://github.com/GoogleChrome/lighthouse
# install chromium browser sudo apt install chromium-browser npm i -g lighthouse # chrome-flags # --no-sandbox is required when user is root # See chrome:cli lighthouse --chrome-flags="--headless --disable-gpu" https://github.com # examples without --chrome-flags to quickly demo options # output format and file name. html is default. json and csv. lighthouse --output json lighthouse --output html --output-path ./report.html # extension is ignored. Result 2 files myfile.report.json and myfile.report.html lighthouse --output json --output html --output-path ./myfile.json lighthouse --output json --output html # saves `./<HOST>_<DATE>.report.json` and `./<HOST>_<DATE>.report.html` lighthouse --output-path=~/mydir/foo.out --save-assets # saves `~/mydir/foo.report.html` # saves `~/mydir/foo-0.trace.json` and `~/mydir/foo-0.devtoolslog.json`
10.4.13. font-awesome npm:font-awesome
npm install font-awesome --savenode_modules/font-awesome/fonts/fontawesome-webfont.woffnode_modules/font-awesome/fonts/fontawesome-webfont.woff2node_modules/font-awesome/fonts/fontawesome-webfont.ttf
10.4.14. prismjs npm:prismjs
npm install prismjs --savenode_modules/prismjs/prism.js- not minified. To minify run
gulp. Default includes these languages:- markup
- xml, html, mathml, svg
- (no term)
- css
- (no term)
- clike
- javascript
- js
components/prism-scss.min.js- more components
- (no term)
themes/prism.css
10.4.15. bootstrap npm:bootstrap
node_modules/node_modules/bootstrap/dist/js/bootstrap.min.jsnode_modules/jquery/dist/jquery.min.jsnode_modules/popper.js/dist/umd/popper.min.js
git clone https://github.com/twbs/bootstrap.git npm install # compile and you will get css and js folders in /dist directory npm run dist # jQuery and popper.js npm install --save jquery popper.js
10.4.16. browser-sync npm:browser-sync
Windows requires node-gyp
It creates localhost:3000 which serves content from proxy server with <script async>...</script> inserted right after <body> HTML.
Refer to this GitHub repo
- Reload CSS without refreshing the page
.pipe( browserSync.stream() )
https
function browsersync() { browserSync.init({ // For more options // @link http://www.browsersync.io/docs/options/ // Project URL. proxy: config.projectURL, // `true` Automatically open the browser with BrowserSync live server. // `false` Stop the browser from automatically opening. open: config.browserAutoOpen, https: { // both absolute and relative paths can be used key: '../../../cs-devops/lndo.site.key', cert: '../../../cs-devops/lndo.site.pem' }, // Inject CSS changes. // Comment it to reload browser for every CSS change. injectChanges: config.injectChanges }); // Use a specific port (instead of the one auto-detected by Browsersync). // port: 7000, }
10.4.17. concurrently npm:concurrently
- https://www.npmjs.com/package/concurrently
- used in Angular
- (no term)
npm install concurrently --saveornpm install -g concurrentlyconcurrently "command1 arg" "command2 arg"- when installed globally
- In package.json
"start": "concurrently \"node server.js\" \"gulp watch\""
10.4.18. concurrent-transform npm:concurrent-transform
npm i -D concurrent-transform
Refer to npm:gulp-image-resize
10.4.19. gulp npm:gulp
https://github.com/gulpjs/gulp/blob/master/docs/README.md
If gulp was previously globally installed, remove it before installing gulp-cli
npm rm --global gulp npm install --global gulp-cli gulp -v # CLI version 2.0.1 # If gulp is installed locally, then # Local version 4.0.0
Then install gulp locally npm install --save-dev gulp@latest
gulp.task( 'default', gulp.parallel( 'styles', 'vendorsJS', 'customJS', 'images', browsersync, function watchFiles() { gulp.watch( config.projectPHPWatchFiles, reload ); // Reload on PHP file changes. gulp.watch( config.styleWatchFiles, gulp.parallel( 'styles' ) ); // Reload on SCSS file changes. gulp.watch( config.vendorJSWatchFiles, gulp.series( 'vendorsJS', reload ) ); // Reload on vendorsJS file changes. gulp.watch( config.customJSWatchFiles, gulp.series( 'customJS', reload ) ); // Reload on customJS file changes. gulp.watch( config.imgSRC, gulp.series( 'images', reload ) ); // Reload on customJS file changes. } ) );
10.4.19.1. Example
gulp runs the default task.
gulp <task> <othertask>
npm install gulp-util --save-dev npm install gulp-coffee --save-dev npm install gulp-concat --save-dev npm install gulp-browserify --save-dev npm install gulp-compass --save-dev npm install jquery --save-dev npm install mustache --save-dev
gulpfile.js
var gulp=require('gulp'), gutil = require('gulp-util'), coffee = require('gulp-coffee'), browserify = require('gulp-browserify'), compase = require('gulp-compass'), concat = require('gulp-concat'); // run ~gulp~ will run this task gulp.task('default', function() { console.log('hello from gulp'); }); // gulp-util, run ~gulp log~ gulp.task('log', function() { gutil.log('Workflow starts'); }); var coffeeSources = ['components/coffee/tagline.coffee']; var jsSources = [ 'components/scripts/1.js', 'components/scripts/2.js' ]; var sassSources = ['components/sass/style.scss']; gulp.task('coffee', function() { gulp.src(coffeeSources) .pipe(coffee({ bare: true }) .on('error', gutil.log)) .pipe(gulp.dest('components/scripts')); }); gulp.task('js', ['coffee'], function() { gulp.src(jsSources) .pipe(concat('script.js')) .pipe(gulp.dest('builds/development/js')); }); gulp.task('compass', function() { gulp.src(sassSources) .pipe(compoass({ sass: 'components/sass', image: 'builds/development/images', style: 'expanded' }) .on('error', gutil.log) ) .pipe(gulp.dest('builds/development/css')); }); gulp.task('watch', function() { gulp.watch(coffeeSources, ['coffee']); });
10.4.19.2. Pass parameters from CLI to Gulp
//region Load CLI Parameters to Gulp // e.g. `gulp task1 --a 123 --b "my string" --c` // arg = { "a": "123", "b": "my string", "c": true } const arg = (argList => { let arg = {}, a, opt, thisOpt, curOpt; for (a = 0; a < argList.length; a++) { thisOpt = argList[a].trim(); opt = thisOpt.replace(/^\-+/, ''); if (opt === thisOpt) { // argument value if (curOpt) arg[curOpt] = opt; curOpt = null; } else { // argument name curOpt = opt; arg[curOpt] = true; } } return arg; })(process.argv); //endregion
10.4.19.3. gulp.src, gulp.dest
gulp.src(globs[, options])- string or array. node-glob
- options to pass to node-glob
negate
// negate can be used gulp.src(['client/*.js', '!client/b*.js', 'client/bad.js'], function() {});
Multiple files
var files = [ 'node_modules/jquery/dist/jquery.min.js', 'node_modules/bootstrap/dist/js/bootstrap.min.js', 'node_modules/popper.js/dist/umd/popper.min.js' ]; gulp.task('vendorjscopy', function() { return gulp.src(files) .pipe(gulp.dest('./public/js') });
Only one
baseis possible for an array of files/* some/path/example/app/js/app.js some/path/example/vendor/js/vendor.js some/path/example/vendor/lib/js/lib.js */ gulp.src('some/path/**/js/*.js') .pipe(gulp.dest('output')); // base is anything before the first ** which is some/path // result /* output/example/app/js/app.js output/example/vendor/js/vendor.js output/example/vendor/lib/js/lib.js */ // hard set base gulp.src('some/path/**/js/*.js', {base:'.'}) .pipe(gulp.dest('output')); // result /* output/some/path/example/app/js/app.js output/some/path/example/vendor/js/vendor.js output/some/path/example/vendor/lib/js/lib.js */ // Does not work! gulp.src( ['source1/examples/**/*.html', 'source2/examples/**/*.html'], { base: ['source1/', 'source2/']} // Doesn't work. Needs to be a string. ).pipe(gulp.dest('dist')); // Instead var merge = require('merge-stream'); gulp.task('default', function() { merge(gulp.src('source1/examples/**/*.html', {base: 'source1/'}), gulp.src('source2/examples/**/*.html', {base: 'source2/'})) .pipe(gulp.dest('dist')); });
- current working directory
10.4.19.4. gulp-debug
See what files are run through Gulp pipeline
10.4.19.5. gulp-plumber
npm install --save-dev gulp-plumber
By design, Node stream will stop accepting incoming data, if error event was raised. You can see it in stream.js:103 - cleanup function will deattach ondata handler from source (which in our case is gulp.src) and coffee plugin will stop receiving files although, the rest of the files can be compiled.
gulp.src('coffee/**/*.coffee') .pipe(gulpPrefixer('// Copyright 2014 (C) Aswesome company')) .on('error', gutil.log) .pipe(coffee()) .on('error', gutil.log) .pipe(gulp.dest('js/'));
Use gulp-plumber
var plumber = require('gulp-plumber'); var coffee = require('gulp-coffee'); gulp.src('./src/*.ext') .pipe(plumber()) .pipe(coffee()) .pipe(gulp.dest('./dist'));
10.4.19.6. gulp-remember
var gulp = require('gulp'), header = require('gulp-header'), footer = require('gulp-footer'), concat = require('gulp-concat'), cache = require('gulp-cached'), remember = require('gulp-remember'); var scriptsGlob = 'src/**/*.js'; gulp.task('scripts', function () { return gulp.src(scriptsGlob) .pipe(cache('scripts')) // only pass through changed files, 'scripts' is any cache name of your choice .pipe(header('(function () {')) // do special things to the changed files... .pipe(footer('})();')) // for example, add a stupid-simple module wrap to each file .pipe(remember('scripts')) // add back all files (initially in gulp.src) to the stream. 'scripts' is any cache name of your choice .pipe(concat('app.js')) // do things that require all files .pipe(gulp.dest('public/')) });
10.4.19.7. gulp-cache
refer to gulp-remember
10.4.19.8. gulp-sort
Sort files in stream by path or any custom sort comparator
npm install gulp-sort --save-dev
var sort = require('gulp-sort'); // default sort gulp.src('./src/js/**/*.js') .pipe(sort()) .pipe(gulp.dest('./build/js')); // pass in a custom comparator function gulp.src('./src/js/**/*.js') .pipe(sort(customComparator)) .pipe(gulp.dest('./build/js')); // sort descending gulp.src('./src/js/**/*.js') .pipe(sort({ asc: false })) .pipe(gulp.dest('./build/js')); // sort with a custom comparator gulp.src('./src/js/**/*.js') .pipe(sort({ comparator: function(file1, file2) { if (file1.path.indexOf('build') > -1) { return 1; } if (file2.path.indexOf('build') > -1) { return -1; } return 0; } })) .pipe(gulp.dest('./build/js')); // sort with a custom sort function var stable = require('stable'); gulp.src('./src/js/**/*.js') .pipe(sort({ customSortFn: function(files, comparator) { return stable(files, comparator); } })) .pipe(gulp.dest('./build/js'));
10.4.19.9. gulp-concat
Concat files using operating system's newLine
var concat = require('gulp-concat'); gulp.task('scripts', function() { return gulp.src('./lib/*.js') .pipe(concat('all.js')) .pipe(gulp.dest('./dist/')); }); // in order gulp.task('scripts', function() { return gulp.src(['./lib/file3.js', './lib/file1.js', './lib/file2.js']) .pipe(concat('all.js')) .pipe(gulp.dest('./dist/')); });
10.4.19.10. gulp-rename
var rename = require("gulp-rename"); // rename via string gulp.src("./src/main/text/hello.txt") .pipe(rename("main/text/ciao/goodbye.md")) .pipe(gulp.dest("./dist")); // ./dist/main/text/ciao/goodbye.md // rename via function gulp.src("./src/**/hello.txt") .pipe(rename(function (path) { path.dirname += "/ciao"; path.basename += "-goodbye"; path.extname = ".md"; })) .pipe(gulp.dest("./dist")); // ./dist/main/text/ciao/hello-goodbye.md // rename via hash gulp.src("./src/main/text/hello.txt", { base: process.cwd() }) .pipe(rename({ dirname: "main/text/ciao", basename: "aloha", prefix: "bonjour-", suffix: "-hola", extname: ".md" })) .pipe(gulp.dest("./dist")); // ./dist/main/text/ciao/bonjour-aloha-hola.md
Rename dest file name Any source *-gulp.css will have dest file *-gulp-build.css
var gulp = require('gulp'); gulp.task('autoprefixer', function () { var postcss = require('gulp-postcss'); var sourcemaps = require('gulp-sourcemaps'); var rename = require('gulp-rename'); var autoprefixer = require('autoprefixer'); return gulp.src('**/*-gulp.css', {base: '.'}) .pipe(sourcemaps.init()) .pipe(postcss([autoprefixer()])) .pipe(sourcemaps.write('.')) .pipe(rename(function (path) { path.basename += '-build'; })) .pipe(gulp.dest('.')); });
10.4.19.11. gulp-sourcemaps
- https://www.npmjs.com/package/gulp-sourcemaps
- Another sourcemap file (endfile.css.map) is generated on top of the result file (endfile.css)
- All plugins between
sourcemaps.init()andsourcemaps.write()need to have support for gulp-sourcemaps - the path is relative to the destination
var gulp = require('gulp'); var plugin1 = require('gulp-plugin1'); var plugin2 = require('gulp-plugin2'); var sourcemaps = require('gulp-sourcemaps'); gulp.task('javascript', function() { gulp.src('src/**/*.js') .pipe(sourcemaps.init()) .pipe(plugin1()) .pipe(plugin2()) .pipe(sourcemaps.write()) .pipe(gulp.dest('dist')); });
10.4.19.12. gulp-filter
const gulp = require('gulp'); const uglify = require('gulp-uglify'); const filter = require('gulp-filter'); gulp.task('default', () => { // Create filter instance inside task function const f = filter(['**', '!*src/vendor'], {restore: true}); return gulp.src('src/**/*.js') // Filter a subset of the files .pipe(f) // Run them through a plugin .pipe(uglify()) // Bring back the previously filtered out files (optional) .pipe(f.restore) .pipe(gulp.dest('dist')); });
10.4.19.13. gulp-sass npm:gulp-sass
Doc Options are the same as node-sass
var scssStream = gulp.src( config.styleSRC ) .pipe( sourcemaps.init() ) .pipe( sass({ errLogToConsole: config.errLogToConsole, outputStyle: config.outputStyle, precision: config.precision }) ) .on( 'error', sass.logError ) .pipe(concat('app.scss'));
- Add extra path for
@import
includePathsis related togulpfile.jsor the file wheresassis run. Default is empty array[]- Say
assets/scss/main.scsshas@import subfolder/a.scss- If file exists
assets/scss/subfolder/a.scss, use it. If not, go to next step - If file exists
assets/scss/supporting/fallback/subfolder/a.scss, use it
- If file exists
var scssStream = gulp.src( config.styleSRC ) .pipe( sourcemaps.init() ) .pipe( sass({ errLogToConsole: config.errLogToConsole, outputStyle: config.outputStyle, precision: config.precision, includePaths: ['assets/scss/supporting/fallback/'] }) ) .on( 'error', sass.logError ) .pipe(concat('app.scss'));
10.4.19.14. gulp-merge-media-queries
var mmq = require('gulp-merge-media-queries'); gulp.task('mmq', function () { gulp.src('src/**/*.css') .pipe(mmq({ log: true })) .pipe(gulp.dest('dist')); });
10.4.19.15. gulp-line-ending-corrector
It converts CRLF \r\n to LF \n
var lec = require( 'gulp-line-ending-corrector' ); // .pipe(lec()) // .pipe(lec({verbose:true, eolc: 'LF', encoding:'utf8'}))
10.4.19.16. gulp-uglifycss
Minify CSS using UglifyCSS
10.4.19.17. gulp-postcss npm:gulp-postcss Doc
npm i -D autoprefixer npm i -D gulp-postcss
gulp.task('autoprefixer', function () { var postcss = require('gulp-postcss'); var sourcemaps = require('gulp-sourcemaps'); var autoprefixer = require('autoprefixer'); return gulp.src('./src/*.css') .pipe(sourcemaps.init()) .pipe(postcss([ autoprefixer() ])) .pipe(sourcemaps.write('.')) .pipe(gulp.dest('./dest')); });
Load multiple PostCSS plugins, refer to npm:autoprefixer for options
var postcss = require('gulp-postcss'); var gulp = require('gulp'); var autoprefixer = require('autoprefixer'); var cssnano = require('cssnano'); gulp.task('css', function () { var plugins = [ autoprefixer({browsers: ['last 1 version']}), cssnano() ]; return gulp.src('./src/*.css') .pipe(postcss(plugins)) .pipe(gulp.dest('./dest')); });
10.4.19.18. gulp-uglify
It uses UglifyJS2 to minify JavaScript files
10.4.19.19. gulp-babel
https://www.npmjs.com/package/gulp-babel https://babeljs.io/docs/en/options
.pipe(
babel({
presets: [
[
'env', // Preset which compiles ES6 to ES5.
{
targets: { browsers: config.BROWSERS_LIST } // Target browser list to support.
}
]
]
})
)
10.4.19.20. gulp-responsive
- Installation
- Requires global node-gyp and dev dependency is sharp
- For JPEG, PNG, WebP and TIFF
npm install --save-dev gulp-responsive # you may need to run this if there is still error, but I didn't need it # npm -g install npm@latest # https://github.com/lovell/sharp/issues/615
10.4.19.21. gulp-image-resize npm:gulp-image-resize
# make sure imagemagick is installed apt list -a --installed imagemagick # graphicsmagick is said to be faster than imagemagick apt install graphicsmagick npm install --save-dev gulp-image-resize # Imagick config # - By default, it prevents the conversion from PDF to PNG. Comment out this line to relax this policy <policy domain="coder" rights="none" pattern="PDF" /> # or <policy domain="coder" rights="read|write" pattern="PDF" />
gulp.task('img-resize-max-width', function () { const imgs = gFilter('**/*.{jpg,JPG,jpeg,JPEG,png,PNG}', {restore: true}); var maxw=1440; return gulp.src('src/**/*') .pipe(imgs) .pipe(gImageResize({ width: maxw, height: 0, imageMagick:true, // default uses graphicmagick interlace: true })) .pipe(imgs.restore) .pipe(gulp.dest('dist')); });
var parallel = require("concurrent-transform"); var os = require("os"); gulp.task("parallel", function () { gulp.src("src/**/*.{jpg,png}") .pipe(parallel( imageResize({ width : 100 }), os.cpus().length )) .pipe(gulp.dest("dist")); });
10.4.19.22. gulp-imagemin
- https://www.npmjs.com/package/gulp-imagemin
npm i -D gulp-imagemin- Come with 4 lossless plugins gifsicle, mozjpeg (used jpegtran before), optipng, svgo
10.4.19.23. gulp-svg-sprite
https://github.com/jkphl/svg-sprite https://github.com/jkphl/gulp-svg-sprite
npm i -D gulp-svg-sprite
There're 5 modes :: css, view, defs, symbol, stack
10.4.19.24. gulp-notify
It sends notification on operating system level.
.pipe( notify({ message: 'TASK: "styles" Completed!', onLast: true }) );
10.4.20. grunt
Install grunt globally and locally
npm install grunt-cli -g npm install grunt-cli --save-dev npm install grunt-contrib-jshint --save-dev npm install grunt-contrib-less --save-dev npm install grunt-autoprefixer --save-dev npm install grunt-browserify --save-dev npm install grunt-contrib-watch --save-dev npm install jquery
// GruntFile.js module.exports = function(grunt) { grunt.initConfig({ jshint: { files: ["*.js", "lib/*.js", "test/*.js"], options: { esnext: true, globals: { jQuery: true } } }, less: { production: { files: { "public/css/style.css": ["less/*.less"] } } }, autoprefixer: { single_file: { src: "public/css/style.css", dest: "public/css/style.css" } }, browserify: { client: { src: ["app-client.js"], dest: "public/js/bundle.js" } }, watch: { css: { files: ["less/*.less"], tasks: ["css"] }, scripts: { files: ["app-client.js", "lib/*.js"], tasks: ["jshint", "browserify"] } } }); grunt.loadNpmTasks("grunt-contrib-jshint"); grunt.loadNpmTasks("grunt-contrib-less"); grunt.loadNpmTasks("grunt-autoprefixer"); grunt.loadNpmTasks("grunt-browserify"); grunt.loadNpmTasks("grunt-contrib-watch"); grunt.registerTask("css", ["less", "autoprefixer"]); grunt.registerTask("js", ["browserify"]); grunt.registerTask("default", ["jshint", "css", "js"]); }; // ./app-client.js var $ = require("jquery"); var printTerms = require("./lib/printTerms"); // jQuery code // run grunt grunt // open another session and run grunt watch
10.4.21. webpack
10.4.21.1. Basics
mkdir webpack-demo
cd webpack-demo
npm init -y
npm install webpack webpack-cli --save-dev
10.4.21.2. Multiple bundles
entry: {
app: 'src/app.ts',
vendor: 'src/vendor.ts'
},
output: {
filename: '[name].js'
}
10.4.21.3. Rules
module: {
rules: [
{
test: /\.ts$/,
loader: 'awesome-typescript-loader'
},
{
test: /\.css$/,
loaders: 'style-loader!css-loader'
}
]
}
10.4.21.4. Plugins
const { UglifyJsPlugin } = require('webpack').optimize; plugins: [ new UglifyJsPlugin({ // config options }) ]
Webpack, the plugins, and the loaders are also installed as packages. They are listed in the updated packages.json.
- terser-webpack-plugin
- https://github.com/webpack-contrib/terser-webpack-plugin
npm i terser-webpack-plugin -Dwebpack.config.js
const TerserPlugin = require("terser-webpack-plugin"); module.exports = { optimization: { minimize: true, minimizer: [new TerserPlugin({ terserOptions: { keep_fnames: true, }, })], }, };
10.4.21.5. resolve
Most import statements don't mention the extension. Tell Webpack to resolve extension-less file requests by looking for matching files with .ts extension or .js extension.
"resolve": { "extensions": [ ".ts", ".js" ], "modules": [ "./node_modules", "./node_modules" ], "symlinks": true }
10.4.21.6. webpack.(common|dev|prod).js
webpack.common.js
var webpack = require('webpack'); var HtmlWebpackPlugin = require('html-webpack-plugin'); var ExtractTextPlugin = require('extract-text-webpack-plugin'); var helpers = require('./helpers'); module.exports = { entry: { 'polyfills': './src/polyfills.ts', 'vendor': './src/vendor.ts', 'app': './src/main.ts' }, resolve: { extensions: ['.ts', '.js'] }, module: { rules: [ { test: /\.ts$/, loaders: [ { loader: 'awesome-typescript-loader', options: { configFileName: helpers.root('src', 'tsconfig.json') } } , 'angular2-template-loader' ] }, { test: /\.html$/, loader: 'html-loader' }, { test: /\.(png|jpe?g|gif|svg|woff|woff2|ttf|eot|ico)$/, loader: 'file-loader?name=assets/[name].[hash].[ext]' }, { test: /\.css$/, exclude: helpers.root('src', 'app'), loader: ExtractTextPlugin.extract({ fallbackLoader: 'style-loader', loader: 'css-loader?sourceMap' }) }, { test: /\.css$/, include: helpers.root('src', 'app'), loader: 'raw-loader' } ] }, plugins: [ // Workaround for angular/angular#11580 new webpack.ContextReplacementPlugin( // The (\\|\/) piece accounts for path separators in *nix and Windows /angular(\\|\/)core(\\|\/)@angular/, helpers.root('./src'), // location of your src {} // a map of your routes ), new webpack.optimize.CommonsChunkPlugin({ name: ['app', 'vendor', 'polyfills'] }), new HtmlWebpackPlugin({ template: 'src/index.html' }) ] };
webpack.dev.js extends webpack.common.js
var webpackMerge = require('webpack-merge'); var ExtractTextPlugin = require('extract-text-webpack-plugin'); var commonConfig = require('./webpack.common.js'); var helpers = require('./helpers'); module.exports = webpackMerge(commonConfig, { devtool: 'cheap-module-eval-source-map', output: { path: helpers.root('dist'), publicPath: '/', filename: '[name].js', chunkFilename: '[id].chunk.js' }, plugins: [ new ExtractTextPlugin('[name].css') ], devServer: { historyApiFallback: true, stats: 'minimal' } }); #+END_EXAMPLE The HtmlWebpackPlugin, added in webpack.common.js, uses the ~publicPath~ and ~filename~ in ~output~ to generate appropriate <script> and <link> tags into the index.html. ~output~ bundles in the dist folder, the dev server keeps all bundles in memory; it doesn't write them to disk. You won't find any files in the dist folder, at least not any generated from this development build. Until the environment variable is set to production (webpack.DefinePlugin) webpack.prod.js #+BEGIN_EXAMPLE var webpack = require('webpack'); var webpackMerge = require('webpack-merge'); var ExtractTextPlugin = require('extract-text-webpack-plugin'); var commonConfig = require('./webpack.common.js'); var helpers = require('./helpers'); const ENV = process.env.NODE_ENV = process.env.ENV = 'production'; module.exports = webpackMerge(commonConfig, { devtool: 'source-map', output: { path: helpers.root('dist'), publicPath: '/', filename: '[name].[hash].js', chunkFilename: '[id].[hash].chunk.js' }, plugins: [ new webpack.NoEmitOnErrorsPlugin(), new webpack.optimize.UglifyJsPlugin({ // https://github.com/angular/angular/issues/10618 mangle: { keep_fnames: true } }), new ExtractTextPlugin('[name].[hash].css'), new webpack.DefinePlugin({ 'process.env': { 'ENV': JSON.stringify(ENV) } }), new webpack.LoaderOptionsPlugin({ htmlLoader: { minimize: false // workaround for ng2 } }) ] });
Angular CLI set ENV in this way. The EnvironmentPlugin is shorthand for using the DefinePlugin on process.env keys.
const { EnvironmentPlugin } = require('webpack'); "plugins": [ new EnvironmentPlugin({ "NODE_ENV": "production" }), ]
webpack.prod.js addtional plugins
- NoEmitOnErrorsPlugin
- stops the build if there is an error.
- UglifyJsPlugin
- minifies the bundles.
- ExtractTextPlugin
- extracts embedded css as external files, adding cache-busting hash to the filename.
- DefinePlugin
- use to define environment variables that you can reference within the application.
- LoaderOptionsPlugins
- to override options of certain loaders.
10.4.21.7. Use of webpack.(common|dev|prod).js
src/index.html
<!DOCTYPE html> <html> <head> <base href="/"> <title>Angular With Webpack</title> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> </head> <body> <my-app>Loading...</my-app> </body> </html>
src/main.ts
import { platformBrowserDynamic } from '@angular/platform-browser-dynamic'; import { enableProdMode } from '@angular/core'; import { AppModule } from './app/app.module'; if (process.env.ENV === 'production') { enableProdMode(); } platformBrowserDynamic().bootstrapModule(AppModule);
polyfills.ts is a good place to configure browser environment for production or development
import 'core-js/es6'; import 'core-js/es7/reflect'; require('zone.js/dist/zone'); if (process.env.ENV === 'production') { // Production } else { // Development and test Error['stackTraceLimit'] = Infinity; require('zone.js/dist/long-stack-trace-zone'); }
src/vendor.ts
// Angular import '@angular/platform-browser'; import '@angular/platform-browser-dynamic'; import '@angular/core'; import '@angular/common'; import '@angular/http'; import '@angular/router'; // RxJS import 'rxjs'; // Other vendors for example jQuery, Lodash or Bootstrap // You can import js, ts, css, sass, ...
Highlights
- There are no <script> or <link> tags in the index.html. The HtmlWebpackPlugin inserts them dynamically at runtime
- The AppComponent in app.component.ts imports the application-wide css with a simple import statement
- The AppComponent itself has its own html template and css file. WebPack loads them with calls to require(). Webpack stashes those component-scoped files in the app.js bundle too. You don't see those calls in the source code; they're added behind the scenes by the angular2-template-loader plug-in
- The vendor.ts consists of vendor dependency import statements that drive the vendor.js bundle. The application imports these modules too; they'd be duplicated in the app.js bundle if the CommonsChunkPlugin hadn't detected the overlap and removed them from app.js
10.4.22. PostCSS, postcss-cli cssnano autoprefixer
All PostCSS plugins, online catalog
Install npm package for a specific build tool e.g. npm:gulp-postcss And install individual PostCSS plugin which is separate npm package
npm:postcss:options Most PostCSS runners accept two parameters:
- An array of plugins.
- An object of options.
Common options:
- syntax: an object providing a syntax parser and a stringifier.
- parser: a special syntax parser (for example, SCSS).
- stringifier: a special syntax output generator (for example, Midas).
- map: source map options.
- from: the input file name (most runners set it automatically).
- to: the output file name (most runners set it automatically).
#+BEGIN_SRC js var gulp = require('gulp'); var postcss = require('gulp-postcss'); var nested = require('postcss-nested'); var sugarss = require('sugarss');
gulp.task('default', function () { var plugins = [nested]; return gulp.src('in.sss') .pipe(postcss(plugins, { parser: sugarss })) .pipe(gulp.dest('out')); }); #+END_SRC js
10.4.22.1. postcss-cli npm:postcss-cli
npm i -g|-D postcss-cli
npm i postcss-cli autoprefixer npx postcss *.css --use autoprefixer -d build/
10.4.22.2. autoprefixer npm:autoprefixer
Required by npm:gulp-postcss
It doesn't remove duplicate css rules and removes unwanted prefixes in source files.
- browserslist npm:postcss:browerslist
package.json
https://github.com/ai/browserslist#queries Default
{ ..., "browserslist": [ "> 0.5%", "last 2 versions", "Firefox ESR", "not dead" ], ... } - Comment to turn off prefixing
Autoprefixer is disabled for a specific rule
ba { transition: 1s; /* it will be prefixed */ } b { /* autoprefixer: off */ transition: 1s; /* it will not be prefixed */ }/* autoprefixer: off */ @supports (transition: all) { /* autoprefixer: on */ a { /* autoprefixer: off */ } }For Sass, add ! so that
/*! autoprefixer: off */ - Options
autoprefixer({ cascade: false })
- browsers (array)
- list of browsers query (like last 2 versions), which are supported in your project. We recommend to use browserslist config or browserslist key in package.json, rather than this option to share browsers with other tools. See Browserslist docs for available queries and default value.
- env (string)
- environment for Browserslist.
- cascade (boolean)
- should Autoprefixer use Visual Cascade, if CSS is uncompressed. Default: true
- add (boolean)
- should Autoprefixer add prefixes. Default is true.
- remove (boolean)
- should Autoprefixer [remove outdated] prefixes. Default is true.
- supports (boolean)
- should Autoprefixer add prefixes for @supports parameters. Default is true.
- flexbox (boolean|string)
- should Autoprefixer add prefixes for flexbox properties. With "no-2009" value Autoprefixer will add prefixes only for final and IE versions of specification. Default is true.
- grid (boolean)
- should Autoprefixer add IE prefixes for Grid Layout properties. Default is false.
- stats (object)
- custom usage statistics for > 10% in my stats browsers query.
Plugin object has info() method for debugging purpose.
You can use PostCSS processor to process several CSS files to increase performance.
- Debug
Show selected browsers and css properties will be prefixed npx autoprefixer –info
10.4.22.3. cssnano npm:cssnano
Build on top of PostCSS, minify css
10.4.22.4. precss npm:precss
Use Sass-like markup and staged CSS features in .css file. npm install precss –save-dev
10.4.23. co-fs, co nodejs:co-fs nodejs:co
Default fs doesn't support Promise or Generator Function
npm install co-fs
npm install co
var fs = require('co-fs');
co(function* () {
var data = yield fs.readFile('./data1.json', 'utf-8');
// fs.readFile is a promise by using co-fs
console.log(data);
});
Before co@4.0.0, co() returned a thunk, which is a callback, after it returns a promise Chain
co(function* () {
var result = yield Promise.resolve(true);
return result;
})
.then(function (value) {
console.log(value);
}, function (err) {
console.error(err.stack);
});
Convert co-generator function into a regular function which returns a promise
var fn = co.wrap(function* (val) {
return yield Promise.resolve(val);
});
fn(true).then(function (val) {
// do something
});
Yieldables are
- promises
- thunks (functions)
- array (parallel execution)
- objects (parallel execution)
- generators (delegation)
- generator functions (delegation)
Resolve multiple promises in parallel
co(function* () {
var a = Promise.resolve(1);
var b = Promise.resolve(2);
var res = yield [a,b];
}).catch(onerror);
function onerror(err) {
console.error(err.stack);
}
10.4.24. http, https nodejs:http
It's sync request.
10.4.24.1. request
var https = require('https');
var fs = require('fs');
var options = {
hostname: "en.wikipedia.org",
port: 443,
path: "/wiki/George_Washington",
method: "GET"
};
var req = https.request(options, function(res) {
var responseBody = "";
console.log(`Server Status: ${res.statusCode}`);
console.log("Response Headers: %j", res.headers);
res.setEncoding("UTF-8");
res.once("data", function(chunk) {
console.log(chunk);
});
res.on("data", function(chunk) {
console.log(`--chunk-- ${chunk.length}`);
responseBody += chunk;
});
res.on("end", function() {
fs.writeFile("george-w.html", responseBody, function(err) {
if (err) throw err;
});
console.log("File downloaded");
});
});
// Set timeout
req.on('socket', function(socket) {
socket.setTimeout(8000);
socket.on('timeout', () => {
req.abort();
console.log('timeout');
});
});
req.on("error", function(err) {
if (err.code === 'ECONNRESET') {
console.log("Timeout occurs");
}
console.log(`problem: ${err.message}`);
});
req.end();
10.4.24.2. server
Simple HTTP server which gives responses
var http = require('http');
var fs = require('fs');
var path = require('path');
var jsonFile = require('./data/inventory'); //inventory.json
// jsonFile is a json object, not string
var server = http
.createServer(function (req, res) {
if (req.url === '/') {
fs.readFile("./public/index.html", "UTF-8", function (err, html) {
res.writeHead(200, {"Content-Type": "text/html"});
// res.write(html); res.end();
res.end(html);
});
}
else if (req.url.match(/.css$/)) {
var cssPath = path.join(__dirname, 'public', req.url);
var fileStream = fs.createReadStream(cssPath, "UTF-8");
res.writeHead(200, {"Content-Type": "text/css"});
fileStream.pipe(res);
}
else if (req.url.match(/.jpg$/)) {
var imgPath = path.join(__dirname, 'public', req.url);
var imgStream = fs.createReadStream(imgPath); // binary
res.writeHead(200, {"Content-Type": "image/jpeg"});
imgStream.pipe(res);
}
else if (req.url === 'getallorders') {
res.writeHead(200, {"Content-Type": "text/json"});
res.end(JSON.stringify(jsonFile));
}
else if (req.url === 'instock') {
listInStock(res);
}
else if (req.url === 'form') {
if (res.method === "GET") {
res.writeHead(200, {"Content-Type": "text/html"});
fs.createReadStream("./public/form.html", "UTF-8").pipe(res);
}
else if (req.method === "POST") {
var body = "";
// body will be a url encoded string with key-value pairs
req.on("data", function(chunk) {
body += chunk;
});
req.on("end", function() {
res.writeHead(200, {"Content-Type": "text/html"});
res.end(`${body}`);
});
}
}
else {
res.writeHead(200, {"Content-Type": "text/plain"});
res.end("404 File Not Found");
}
})
.listen(3000);
function listInStock(res) {
var inStock = jsonFile.filter(function(item) {
return item.avail === "In Stock";
});
res.end(JSON.stringify(inStock));
}
10.4.25. cors
You don't need to config Nginx!
npm install cors
var express = require('express');
var cors = require('cors');
var app = express();
// Enable CORS for all routes/requests
app.use(cors());
app.get('/products/:id', function (req, res, next) {
res.json({msg: 'This is CORS-enabled for all origins!'});
});
// Enable for one route
// Don't put cors() in app.route()
app.get('/products/:id', cors(corOptions), function (req, res, next) {
res.json({msg: 'This is CORS-enabled for all origins!'})
})
// Options
var corsOptions = {
origin: 'http://example.com',
optionsSuccessStatus: 200
}
// Multiple domains
var whitelist = ['http://example1.com', 'http://example2.com']
var corsOptions = {
origin: function (origin, callback) {
if (whitelist.indexOf(origin) !== -1) {
callback(null, true)
} else {
callback(new Error('Not allowed by CORS'))
}
}
}
// use the same cors(corsOptions)
10.4.26. xml2js
https://github.com/Leonidas-from-XIV/node-xml2js
npm install --save xml2js
var parseString = require('xml2js').parseString;
var xml = "<root>Hello xml2js!</root>"
parseString(xml, function (err, result) {
console.dir(result);
});
10.4.27. casual nodejs:casual
10.4.28. crypto - Built-in nodejs:crypto
10.4.28.1. HMAC
SSL certificate changes from SHA1 to SHA256 which is SHA2 algorithm.
One ASCII character is 8 bits. One UTF-8 character is between 8 bits to 32 bits.
One ASCII character can be represented as 2 hex characters (UTF-8, etc.)
Output size of SHA256 is 256 bits, which is 256/8*2 = 64 hex characters Output size of SHA1 is 160, which is 40 hex characters
The secret/key of SHA256 should at least contain 256 bits = 256/8 bytes = 32 bytes. But I use 128 bits.. Which is 128/8*2 = 32 hex characters.
var crypto = require('crypto') , text = 'hello' , key = 'abcdeg' // better to have 32 hex characters, but length could be any , hash let hmac = crypto.createHmac('sha256', key); hmac.write(text); hmac.end(); let hash = hmac.read().toString('hex');
10.4.28.2. encrypt, decrypt
Refer to php:openssl_decrypt
All AES, regardless 128/256 or key length, the output size depends on the input size (floor(Input size / 16 bytes) + 1) * 16 bytes
const crypto = require('crypto'); const KEYAES = '64 hex characters' const ALGORITHM = 'aes-256-cbc'; // store this keyAES256.toString('hex') // convert hex to bytes // keyAES256 = new buffer(keyAES256, 'hex'); function encrypt(plain_text) { var iv = crypto.randomBytes(16); // 16 text characters, 16 bytes or 32 hex characters // iv should be different every time var keyAES = new Buffer(KEYAES, 'hex'); // var keyAES = crypto.randomBytes(32); // 32 text characters, 32 bytes or 64 hex characters // global KEYAES is keyAES.toString('hex') var cipher = crypto.createCipheriv(ALGORITHM, keyAES, iv); cipher.write('text to encrypt'); cipher.end(); var cipher_text = cipher.read().toString('hex'); return cipher_text + '$' + iv.toString('hex'); } function decrypt(cipher_text) { var cipher_blob = cipher_text.split("$"); var ct = cipher_blob[0]; var iv = new Buffer(cipher_blob[1], 'hex'); // get iv var keyAES = new Buffer(KEYAES, 'hex'); decryptor = crypto.createDecipheriv(ALGORITHM, keyAES, iv); decryptor.write(ct, 'hex'); decryptor.end(); console.log(decryptor.read().toString('utf-8')); }
10.4.28.3. generate random characters for unique id
let id = require('crypto').randomBytes(10).toString('hex')
10.4.29. body-parser
parse POST data. Refer to nodejs:express
npm install body-parser
var express = require('express');
10.4.30. httpster
npm install httpster -g httpster -p 3000 -d ./public/ # Or httpster is installed locally ./node_modules/httpster/bin/httpster -p 3000 -d ./public/
10.4.31. express nodejs:express
10.4.31.1. Sample
// my-express.js var express = require('express'); var app = express(); var bodyParser = require('body-parser'); app.set('port', process.env.PORT || 3000); var skierTerms = [{},{}]; // javascript array/object. global to the project/app. Only reset when app is restarted. app.set('appData', skierTerms); // app.use() adds middlewares which run first app.use(bodyParser.json()); // any data sent to this app (API) as .json, the data will be parsed app.use(bodyParser.urlencoded({ extended: true })); // also parse url encoded data // false uses the `querystring` library which means the data could be string or array // true uses the `qs` library which means the data could be any type // log every request app.use(function(req, res, next) { console.log(`${req.method} request for '${req.url}' - ${JSON.stringify(req.body)}`); next(); // eventually, res.send('output');, so here is next }); // default is index.html, server static files app.use(express.static(__dirname)); app.use('/message', function(req, res) { console.log('user requested endpoint'); res.send('response is hello'); }); app.use(require('./routes/index')); app.use(require('./routes/dict-api')); app.listen(app.get('port'), function() { // optional callback }); // visit localhost:3000/message // node my-express.js // PORT=4000 node my-express.js
// ./routes/dict-api.js var express = require('express'); var router = express.Router(); router.use(bodyParser.json()); // any data sent to this app (API) as .json, the data will be parsed router.use(bodyParser.urlencoded({ extended: false })); // also parse url encoded data // same domain router.get('/dict-api', function(req, res) { var skierTerms = req.app.get('appData'); res.json(skierTerms); // send json variable, no need to stringify }); router.post('/dict-api', function(req, res) { var skierTerms = req.app.get('appData'); skierTerms.push(req.body); req.app.set('appData', skierTerms); res.json(skierTerms); // send json variable, no need to stringify }); router.delete('dict-api/:term', function(req, res) { var skierTerms = req.app.get('appData'); skierTerms = skierTerms.filter(function(def) { return def.term.toLowerCase() !== req.params.term.toLowerCase(); }); req.app.set('appData', skierTerms); res.json(skierTerms); }); module.exports = router;
10.4.31.2. Routing
- Callbacks
Instead of
next()which skips to the next callback,next('route')can bypass all the remaining route callbacks of the current app.METHOD() or router.METHOD().next('router')to skip the rest of the router's middleware functions the router instanceapp.get('/example/a', function (req, res) { res.send('Hello from A!') }) app.get('/example/b', function (req, res, next) { console.log('the response will be sent by the next function ...') next() }, function (req, res) { res.send('Hello from B!') }) var cb0 = function (req, res, next) { console.log('CB0') next() } var cb1 = function (req, res, next) { console.log('CB1') next() } var cb2 = function (req, res) { res.send('Hello from C!') } app.get('/example/c', [cb0, cb1, cb2]) var cb0 = function (req, res, next) { console.log('CB0') next() } var cb1 = function (req, res, next) { console.log('CB1') next() } app.get('/example/d', [cb0, cb1], function (req, res, next) { console.log('the response will be sent by the next function ...') next() }, function (req, res) { res.send('Hello from D!') })
- Response methods
- res.send res.sendFile
// Content-Type is automatically sent when res.send is used res.send(new Buffer('whoop')); res.send({ some: 'json' }); // Array or Object, Express responds with JSON res.send('<p>some html</p>'); // Content-Type text/html res.status(404).send('Sorry, we cannot find that!'); res.status(500).send({ error: 'something blew up' }); // Send an octet stream :: http://expressjs.com/en/4x/api.html#res.sendFile // res.sendFile(path [, options] [, fn]) res.sendFile(__dirname + '/views/index.html' );
- res.write res.end
Quickly ends the response without any data.
// send multiple responses using res.write res.write('1'); res.write('2'); res.setHeader('Content-Type', 'text/html'); res.end(); // res.status(404).end();
- res.send res.sendFile
- Parameters
// Route path: /users/:userId/books/:bookId // Request URL: http://localhost:3000/users/34/books/8989 // req.params: { "userId": "34", "bookId": "8989" } app.get('/users/:userId/books/:bookId', function (req, res) { res.send(req.params) }) // Route path: /user/:userId(\d+) // Request URL: http://localhost:3000/user/42 // req.params: {"userId": "42"} // Request URL: /mv?id=abc // req.query: {id: "abc"}
The name of route parameters must be made up of “word characters” ([A-Za-z0-9_]).
-and.are literal characters - and .For using
*as regex, use{0,}for Express 4.x, use*for Express 5.
10.4.31.3. Error-handling middleware
Error-handling middleware are defined last.
A built-in error handler is added at the end.
If next(err) is called and there're no custom error handler, it'll be handled by the built-in error handler.
In non-production environment (NODE_ENV=production), stack trace is printed.
For a custom error handler, if the headers have already been sent to the client, you'll need to call next(err) to let the built-in error handler to close the connection and fail the request.
next(err) can be only called once otherwise there is an error
function errorHandler(err, req, res, next) { if (res.headersSent) { return next(err) } res.status(500) res.render('error', {error: err}) }
Simple sample:
// 4 arguments app.use(function (err, req, res, next) { console.error(err.stack); res.status(500).send('Error!'); })
Multiple error handlers
// /index.js app.use(logErrors) app.use(clientErrorHandler) app.use(errorHandler) function logErrors (err, req, res, next) { console.error(err.stack) next(err) } function clientErrorHandler (err, req, res, next) { if (req.xhr) { res.status(500).send({ error: 'Something failed!' }) } else { next(err) } } function errorHandler (err, req, res, next) { res.status(500) res.render('error', { error: err }) }
10.4.31.4. Built-in middleware
10.4.31.5. View Engine (ejs)
npm install ejs --save // app.js app.set('view engine', 'ejs'); app.set('view', './views'); // default app.locals.siteTitle = 'Website Name'; // available to all views // persist through the life of the app // ./routes/index.js router.get('/', function(req,res) { var data = req.app.get('appData'); var pagePhotos = []; data.speakers.forEach(function() { pagePhotos = pagePhotos.concat(item.artwork); }); res.render('index', { pageTitle: 'Home', artwork: pagePhotos, pageID: 'home' }); }); // ./views/index.ejs <!-- HTML Code --> <%= siteTitle %> <%= pageTitle %> <body id="<%= pageID %>"> // if the variable is HTML, output as HTML <%- varHTML %> <% if (typeof pageTitle !== "undefined") { %> <script src="abc.js"></script> <% } %> <% if (artwork.length > 0) { %> <% for (i=0; i< artwork.length; i++) { %> <img src="/images/artwork/<%= artwork[i] %>" alt="Artwork <%= i %>"> <% } %> <% } %> <% include partials/template/header.ejs %> // partial can use the parent view variables
10.4.32. sails
npm install sails -g mkdir sails cd sails // Create a sails project with backend only sails new --no-front-end // create an api sails generate api user // 2 files are created // ./sails/api/controllers/UserController.js // ./sails/api/models/User.js
Modify models/User.js, what attributes a user can have and the datatype?
module.exports = {
attributes: {
name: 'string'
}
}
Start sails, default port is 1337 :: sails lift
Perform CRUD by visiting url
localhost:1337/user/create?name=John localhost:1336/user
10.4.33. koa
ES6
npm install koa
// koa-endpoint-demo.js
var koa = require('koa');
var app = new koa();
app.use(function* () {
this.body = 'hello in the body tag';
});
app.listen(3000);
Run it in harmony
node --harmony koa-endpoint-demo.js
10.4.34. Promise
10.4.34.1. Basics
Promise is native, refer to nodejs:bluebird
var promise = new Promise(function(res, rej) {
resolve();
// or
// reject();
});
promise.then(function() {
console.log('then');
})
.catch(function() {
console.error('failed');
});
10.4.34.2. http server example
const parseString = require('xml2js').parseString;
route.get('/auth/:site/:emailkey', function(req, res) {
try {
getUserInfo(req.params.site, req.params.emailkey, res);
}
catch (e) {
res.json(e.message);
}
});
function getUserInfo(site, emailkey, res) {
httpsRequest(httpsOptions)
.then(function(body) {
parseString(body, (err, result) => {
res.json(result);
});
});
}
function httpsRequest(params) {
return new Promise(function(resolve, reject) {
let req = https.request(params, function(res) {
let body = '';
res.on('data', (chunk) => {
body += chunk;
});
res.on('end', () => {
promisesParser(body)
.then((result) => {
resolve(body);
})
.catch((err) => {
reject(err);
});
});
}); // https.request
req.end();
}); // Promise
}
function promiseParser(string) {
return new Promise(function(resolve, reject) {
parseString(string, function(err, result) {
if (err) {
reject(err);
}
else {
resolve(result);
}
});
});
}
10.4.35. bluebird nodejs:bluebird
// npm install bluebird var fs = require('fs'); var Promise = require('bluebird'); Promise.promisifyAll(fs); // append Async after every fs object (e.g. methods) fs.readFileAsync('./data1.json') .then(JSON.parse) .then(function (val) { console.log(val); }) .catch(SyntaxError, function(e) { console.error("invalid json in file"); }) .catch(function(e) { console.error("unable to read file"); });
10.4.36. lite-server npm:lite-server
- A web server that refreshes when file is updated
- https://www.npmjs.com/package/lite-server
// npm i -g lite-server
10.4.37. node-dev, nodemon, reload
node-dev restart automatically when there's a change
npm install node-dev -g npm install nodemon -g node-dev app.js # vs node app.js # nodemon can watch on a list of file types or ignore a file/folder nodemon -e js,jade app --watch /another/folder/to/watch --ignore feedback.json --ignore aFolder/
In Vagrant, use nodemon -L index.js for --legacy-watch restarting server
Create a WebSocket on the client and reload the browser when Express server is reloaded
npm install -g reload
npm install reload --save
var reload = require('reload');
// app.js
// after app.listen();
reload(server, app);
// In each route, add reload client js file.
router.get('/', function(req, rs) {
res.write('<script src="/reload/reload.js"></script>');
});
10.4.38. jshint
Check javascript syntax
npm install jshint -g jshint app.js // In app.js, insert a comment at top to give extra instructions // Use ES6 standard /* jshint esnext: true */
10.4.39. mongoose nodejs:mongoose
- Refer to jsSandbox:lynda-graphql_essential_training
10.4.40. sequelize nodejs:sequelize
- ORM for MySQL, MariaDB, SQLite, Postgres and MSSQL
- Refer to jsSandbox:lynda-graphql_essential_training
10.4.41. ws
Refer to js:ws for client
npm install ws --save
var WebSocketServer = require('ws');
var wss = new WebSocketServer({port:3000});
wss.on('connection', function(ws) {
wss.on('message', function(msg) {
if (message === 'exit') {
ws.close();
}
else {
// broadcast
wss.clients.forEach(function(client) {
client.send(msg);
});
}
});
ws.send('Welcome to cyber chat');
});
10.4.42. socket.io
npm install socket.io --save
var express = require('express');
var http = require('http'); // socket.io requires http
var app = express();
var server = http.createServer(app).listen(3000);
var io = require('socket.io')(server); // change http server to a socket server
// Or io can be attached later: require('socket.io')();
app.use(express.static('./public'));
// socket is attached later
// io.attach(server);
io.on('connection', function(socket) {
socket.on('chat', function(msg) {
socket.broadcast.emit('message', message);
});
socket.emit('message', 'Welcome to cyber chat');
});
// socket.io-client.min.js
// main.js
var socket = io("http://localhost:3000");
socket.on('disconnect', function() {
// do something
});
socket.on('connect', function() {
// do something
});
socket.on('message', function(msg) {
console.log(msg);
});
socket.emit('chat', input.value);
10.4.43. mocha, chai, nock, rewire, sinon
npm install mocha -g
npm install chai --save-dev
// nock is an http mocking and expectations library
npm install nock --save-dev
// rewire adds special setter and getter to modules to modify their values
// https://www.npmjs.com/package/rewire
npm install rewire --save-dev
npm install sinon --save-dev
// ./test/tools-spec.js
var expect = require('chai').expect;
var tools = require('../lib/tools'); // tools.js
var nock = require('nock');
// Test function printName()
describe('printName()', function() {
// Test 1
it("should print the last name first", function() {
var results = tools.printName({ first: "Li", last: "Banks" });
expect(results).to.equal("Banks, Alex");
});
});
// Test async function loadWiki()
describe('loadWiki()', function() {
this.timeout(5000); // default timeout is 2 seconds
// Test 1
it("Load Abraham Lincon's wiki page", function(done) {
tools.loadWiki({ first: 'Abraham', last: 'Lincooln'}, function(html) {
expect(html).to.be.ok;
done(); // if done() is not run, test will fail
});
});
});
var expect = require('chai').expect;
var tools = require('../lib/tools'); // tools.js
var nock = require('nock');
var rewire = require('rewire');
var order = rewire('../lib/order');
// instead of require(), use rewire() which can change data later
// Test function printName()
describe('printName()', function() {
// Test 1
it("should print the last name first", function() {
var results = tools.printName({ first: "Li", last: "Banks" });
expect(results).to.equal("Banks, Alex");
});
});
// Test async function loadWiki()
describe('loadWiki()', function() {
//this.timeout(5000); // default timeout is 2 seconds
before(function() {
// before runs before all tests in this block
// mock server, save testing time
nock('https://en.wikipedia.org')
.get('/wiki/Abraham_Lincoln')
.reply(200, `--html-code--`);
});
// Test 1
it("Load Abraham Lincon's wiki page", function(done) {
tools.loadWiki({ first: 'Abraham', last: 'Lincooln'}, function(html) {
//expect(html).to.be.ok;
expect(html).to.equal('--html-code--');
done(); // if done() is not run, test will fail
});
});
});
describe('Ordering Items', function() {
beforeEach(function() {
// beforeEach runs before each test in this block
this.testData = [
{sku: "AAA", qty: 10},
{sku: "BBB", qty: 10}
];
// modify data for testing purpose
order.__set__("inventoryData", this.testData);
});
it('order an item when there are enough in stock', function(done) {
order.orderItem('BBB', 3, function() {
// orderItem subtracts 3 from inventory
done();
});
});
});
var expect = require('chai').expect;
var tools = require('../lib/tools'); // tools.js
var nock = require('nock');
var rewire = require('rewire');
var sinon = require('sinon');
var order = rewire('../lib/order');
// instead of require(), use rewire() which can change data later
// Test function printName()
describe('printName()', function() {
// Test 1
it("should print the last name first", function() {
var results = tools.printName({ first: "Li", last: "Banks" });
expect(results).to.equal("Banks, Alex");
});
});
// Test async function loadWiki()
describe('loadWiki()', function() {
//this.timeout(5000); // default timeout is 2 seconds
before(function() {
// before runs before all tests in this block
// mock server, save testing time
nock('https://en.wikipedia.org')
.get('/wiki/Abraham_Lincoln')
.reply(200, `--html-code--`);
});
// Test 1
it("Load Abraham Lincon's wiki page", function(done) {
tools.loadWiki({ first: 'Abraham', last: 'Lincooln'}, function(html) {
//expect(html).to.be.ok;
expect(html).to.equal('--html-code--');
done(); // if done() is not run, test will fail
});
});
});
describe('Ordering Items', function() {
beforeEach(function() {
// beforeEach runs before each test in this block
this.testData = [
{sku: "AAA", qty: 10},
{sku: "BBB", qty: 10}
];
// modify data for testing purpose
order.__set__("inventoryData", this.testData);
this.console = {
// overwrite a function with a fake function
log: sinon.spy()
};
order.__set__('console', this.console);
this.warehouse = {
packageAndShip: sinon.stub().yields(1098765)
};
order.__set__('warehouse', this.warehouse);
});
it('order an item when there are enough in stock', function(done) {
var _this = this;
order.orderItem('BBB', 3, function(done) {
// orderItem subtracts 3 from inventory
expect(_this.console.log.callCount).to.equal(2);
done();
});
});
describe('Warehouse interaction', function() {
beforeEach(function() {
this.callback = sinon.spy();
order.orderItem("BBB", 2, callback);
});
it('receives a tracking number', function() {
expect(this.callback.calledWith(1098765)).to.equal(true);
});
it('calls packageAndShip with the correct sku and quantity', function() {
expect(this.warehouse.packageAndShip.calledWith("BBB",2)).to.equal(true);
});
});
});
// ./lib/tools.js
var https = require('https');
module.exports = {
printName(person) {
return `${person.last}, ${person.first}`;
},
loadWiki(person, callback) {
var url = `https://en.wikipedia.org/wiki/${person.first}_${person.last}`;
https.get(url, function(res) {
var body = "";
res.setEncoding('UTF-8');
res.on("data", function(chunk) {
body += chunk;
});
res.on("end", function() {
callback(body);
});
});
}
};
// ./lib/order.js
var inventoryData = require('../data/inventory');
var warehouse = require('./warehouse');
function findItem(sku) {
var i = inventoryData.map(item => item.sku).indexOf(sku);
if (i === -1) {
console.log(`Item - ${sku} not found`);
return null;
} else {
return inventoryData[i];
}
}
function isInStock(sku, qty) {
var item = findItem(sku);
return item && item.qty >= qty;
}
function order(sku, quantity, complete) {
complete = complete || function () {};
if (isInStock(sku, quantity)) {
console.log(`ordering ${quantity} of item # ${sku}`);
warehouse.packageAndShip(sku, quantity, function (tracking) {
console.log(`order shipped, tracking - ${tracking}`);
complete(tracking);
});
return true;
} else {
console.log(`there are not ${quantity} of item '${sku}' in stock`);
return false;
}
}
module.exports.orderItem = order;
// run mocha
mocha
mocha command not found, add this to package.json
"scripts": {
"test": "mocha -R spec"
}
-R spec is used to console.log inside tests it()
10.4.44. supertest, cheerio
npm install supertest --save-dev
npm install cheerio --save-dev
// ./app.js is an express app
// ./test/app-spec.js
var request = require("supertest");
var expect = require('chai').expect;
var cheerio = require("cheerio");
var rewire = require('rewire');
var app = rewire('../app');
describe("Dictionary App", function () {
it("Loads the home page", function(done) {
request(app).get("/").expect(200).end(function(err, res) {
var $ = cheerio.load(res.text); // jQuery DOM!
var pageHeading = $("body>h1:first-child").text();
expect(pageHeading).to.equal("Skier Dictionary");
done();
});
});
describe("Dictionary API", function () {
beforeEach(function () {
this.defs = [
{
term: "One",
defined: "Term One Defined"
},
{
term: "Two",
defined: "Term Two Defined"
}
];
app.__set__("skierTerms", this.defs);
});
it("GETS dictionary-api", function(done) {
var defs = this.defs;
request(app).get("/dictionary-api").expect(200).end(function(err, res) {
var terms = JSON.parse(res.text);
expect(terms).to.deep.equal(defs);
done();
});
});
it("POSTS dictionary-api", function(done) {
request(app)
.post("/dictionary-api")
.send({ "term": "Three", "defined": "Term Three Defined"})
.expect(200)
.end(done);
});
it("DELETES dictionary-api", function(done) {
request(app)
.delete("/dictionary-api/One")
.expect(200)
.end(done);
});
});
});
10.4.45. istanbul
Code coverage report (testing).
npm install istanbul -g # at ./ istanbul cover _mocha # ./coverage/lconv-report/index.html
10.4.46. svgo nodejs:svgo
// I found windows has permission issue if it's installed globally npm install svgo // optimize the svg ./node_modules/svgo abc.svg // output the svg to data uri URL encoded // enc, unenc, base64 svgo abc.svg --datauri enc // increase precision number of decimal points svgo a.svg --precision=8 // output result STDOUT. if not, the file will be overwritten svgo abc.svg --datauri enc -o -
Usually it's good in IE11 but if it still doesn't work, refer to svg:ie11
AI already makes .svg and you want to move styles to inline svgo abc.svg –enable=inlineStyles –config '{ "plugins": [ { "inlineStyles": { "onlyMatchedOnce": false } }] }'
Move style attribute from each node to attributes :: style="width:…;height:…;" to width="…" height="…" svgo abc.svg –enable=convertStyleToAttrs
–enable=PLUGIN : Enable plugin by name, "–enable={PLUGIN3,PLUGIN4}" for multiple plugins
Batch convert all *.svg files in a folder and output to another folder svgo -f ../path/to/folder/with/svg/files -o ../path/to/folder/with/svg/output
10.4.47. serve
- default port
- 3000
- To change port in command line
serve -s build -p 3001
- (no term)
npm i -g serve
10.5. Custom Module
// my-module.js // Export one property at a time exports.myText = 'hello'; // Variables local to the module will be private // my-square.js, square module // Root of this module is a function // Or you want to export a complete object in one assignment instead of building it one property at a time // Assign it to `module.exports` instead of `exports` module.exports = (width) => { return { area: () => width ** 2 } } // module-demo.js var myModule= require('./my-module.js'); const square = require('./my-square.js'); const mySquare = square(2); console.log(myModule.myText); console.log(`The area of my square is ${mySquare.area()}`); // node module-demo.js
# Initiate the custom module, adding package.json file npm init # --save flag will add/remove dependency to package.js npm install cors --save npm remove cors --save # delete ./node_modules and run this to install all dependencies npm install
Call local functions
var _foo = function () { return ('foo'); } var _bar = function () { return _foo(); } module.exports = { foo: _foo, bar: _bar };
10.5.1. package.json, node-inspector
npm install node-inspector -g- refer to
devDependencies - Run
startinscripts - Run a
scriptinscripts - Property
- main
- a file to include when another user to include the module by
require('ski-dictionary') - devDependencies
npm install --productionor whenNODE_ENVenvironment variable is production, npm will not install modules listed indevDependencies
{
"name": "ski-dictionary",
"version": "1.0.0",
"description": "A collection of skier terms and definitions",
"main": "app.js",
"scripts": {
"predebug": "grunt",
"debug": "open http://localhost:3000 & open http://localhost:8080/debug?port=5858",
"prestart": "grunt",
"start": "node app",
"predev": "grunt",
"dev": "open http://localhost:3000 & node-dev app & grunt watch"
},
"keywords": [
"ski",
"terms",
"dictionary"
],
"author": "Alex Banks",
"license": "MIT",
"dependencies": {
"body-parser": "^1.14.1",
"cors": "^2.7.1",
"express": "^4.13.3",
"jquery": "^2.1.4"
},
"devDependencies": {
"grunt": "^0.4.5",
"grunt-autoprefixer": "^3.0.3",
"grunt-browserify": "^4.0.1",
"grunt-contrib-jshint": "^0.11.3",
"grunt-contrib-less": "^1.0.1",
"grunt-contrib-watch": "^0.6.1"
}
}
10.6. node-gyp node-gyp
Some plugins need node-gyp installed Open PowerShell with admin right
npm install --global --production windows-build-tools npm install --global node-gyp
windows-build-tools downloads and installs VC++ Build Tools and Python 2.7, configuring maching and npm appropriately.
The install is conflict-free meaning that they do not mess with existing installations of VS, C++ Build Tools (C++ runtime libraries) or Python.
10.7. NPM CLI
npm run-script <command> [--silent] [-- <args>...]
10.7.1. run-script nodejs:npm:cli:run-script
npm run-script <command> [--silent] <-- <args>...>]- alias
- without command will return the list of scripts
npm run <a-script-in-scripts>- ">pass custom arguments to the command specified but not the command's pre or post script
- https://docs.npmjs.com/misc/scripts Some of them:
preinstall,[install, postinstall]npm install"install": "node-gyp rebuild"- default if
binding.gypexists in root
pretest,test,posttestnpm testprestop,stop,poststopnpm stopprestart,start,poststartnpm start"start": "node server.js"- default if
server.jsexists in root
prerestart,restart,postrestartnpm restart
npm run <stage>- Run a script from dependencies
10.7.1.1. postinstall
{
"name": "...-...-...",
"author": "...",
"version": "2.0.2",
"license": "GPL-3.0",
"repository": {
"type": "git",
"url": "https://github.com/..."
},
"dependencies": {
...
},
"devDependencies": {
...
},
"scripts": {
"postinstall": "npm run start",
"start": "gulp",
"styles": "gulp styles",
"stylesRTL": "gulp stylesRTL",
"vendorsJS": "gulp vendorsJS",
"customJS": "gulp customJS",
"images": "gulp images",
"clearCache": "gulp clearCache",
"translate": "gulp translate"
}
}
10.7.2. install npm:cli:install
npm i abc- wihtout giving options, current version of npm will auto add it to dependency in package.json
- (no term)
npm i -Deq.npm install --save-devnpm i gist:mygistid- create a Gist on GitHub with package.json
10.8. Increase memory and NODE_OPTIONS nodejs:memory
- Default memory is 512mb. Increase to 1gb
- https://nodejs.org/api/cli.html#cli_node_options_options
- For command line e.g.
npm installNODE_OPTIONS=--max-old-space-size=1024 npm install something
- For node project
node --max_old_space_size=1024 main.js
- For gulp
gulp yourTask --max_old_space_size=1024ornode --max-old-space-size=1024 ./node_modules/.bin/gulp yourTask
- Using a package and build a script
npm install --save-dev increase-memory-limit// ... "scripts": { "fix-memory-limit": "cross-env LIMIT=1024 increase-memory-limit" }, "devDependencies": { "increase-memory-limit": "^1.0.3", "cross-env": "^5.0.5" } // ...
10.9. yarn
10.9.1. Basics
npm install -g yarn mkdir test cd test yarn init # add a devDependency yarn add --dev ava # a yaml file `yarn.lock` is created # install a global package yarn global add rollup
10.10. TS: gulp-util is deprecated
- Older version of a pkg might use gulp-util pkg which is deprecated
npm ls gulp-util- Say
gulp-sassis using it, update it- See installed
gulp-sassversion npm ls gulp-sass- See latest version avaible
npm view gulp-sass versions --json- Update it and update package.json
npm i -D gulp-sassornpm i --save gulp-sass- Reinstall
rm -rf node_modules && rm -f package-lock.json && npm install
- See installed
10.11. TS: pathspec
Command failed: C:\Program Files\Git\mingw64\bin\git.EXE checkout 4.0 error: pathspec '4.0'
Change package.json from "gulp": "gulpjs/gulp#4.0" to "gulp": "4.0"
10.12. TS: Unmet dependencies
rm -rf node_modulesrm package-lock.jsonnpm installnpm cache verifyornpm cache cleanin older NPM version- The problem might be
- Fail to download the package, timed-out.
- Manully install top-level modules
npm i eslink-config-wordpress - or adjust the order of packages in package.json
10.13. TS: ENOSPC: System limit for number of file watchers reached
Increase the inotify max_user_watches limit
For Debian
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
11. Prerender.io
https://github.com/prerender/prerender
Install middleware on .htaccess to rewrite search engine traffic to prerender server
example.com is your site and its .htaccess is
# Change YOUR_TOKEN to your prerender token and uncomment that line if you want to cache urls and view crawl stats # Change http://example.com (at the end of the last RewriteRule) to your website url <IfModule mod_headers.c> #RequestHeader set X-Prerender-Token "YOUR_TOKEN" </IfModule> <IfModule mod_rewrite.c> RewriteEngine On <IfModule mod_proxy_http.c> RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR] RewriteCond %{QUERY_STRING} _escaped_fragment_ # Only proxy the request to Prerender if it's a request for HTML RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/http://example.com/$2 [P,L] </IfModule> </IfModule>
Install prerender on prerender server
git clone https://github.com/prerender/prerender.git
cd prerender
npm install
node server.js
Prerender will now be running on http://localhost:3000. If you wanted to start a web app that ran on say, http://localhost:8000, you can now visit the URL http://localhost:3000/http://localhost:8000 to see how your app would render in Prerender.
:3000/http://:8000 may return something that is 504. Don't worry it will work after Rewrite.
Change options in ./server.js
For better result, add this to every page and search engine will add ?_escaped_fragment_ to the URL
<meta name="fragment" content="!">
- html5 push state
- URL
http://example.com/user/1becomeshttp://example.com/user/1?_escaped_fragment_= - hashbang
http://example.com/#!/user/1becomeshttp://example.com/?_escaped_fragment_=/user/1
12. Social Media
12.1. WhatsApp
https://wa.me/?text=
https://wa.me/15551234567/?text=
- Old
whatsapp://send?text=
$temp_link = get_the_title().' : '.get_permalink().'?utm_source=whatsapp&utm_medium=social&utm_campaign=WhatsappShare'; $temp_link = 'https://wa.me/?text='.urlencode($temp_link);
12.2. Twitter
12.2.1. Compose link with prepopulated Tweet
- https://dev.twitter.com/web/tweet-button/web-intent
https://twitter.com/intent/tweet?- Query Parameters
- text
Hello%20World- url
https%3A%2F%2Fexample.com%2F- hashtags
nature,sunset
- Query Parameters
12.2.2. View a hashtag topic
12.2.3. Recent Tweets
Widget shows png image converted to jpg with 85% quality and black background color. Maybe add a transparent pixel and save the png to truecolor to avoid the conversion.
12.2.4. Twitter card twitter:card
- https://developer.twitter.com/en/docs/tweets/optimize-with-cards/guides/getting-started
- https://cards-dev.twitter.com/validator
- Content is cached by Twitter for 7 days after a link to your page with card markdup has been published in a tweet
12.2.5. TweetDeck twitter:tweetdeck
- https://tweetdeck.twitter.com
- Account
@abcis created byabc@gmail.com,abc@gmail.comis the owner, one account has only one owner- May use
a.bc@gmail.comto create another Twitter account, emails will also send toabc@gmail.com - can link other Twitter accounts as admin or contributor
- Manage password, phone number and login verification settings
- can link other accounts as admin or contributor, but Owner has to authorize
- Tweet, Retweet, DM, like, etc., schedule Tweets, create lists and build collections
- permissions like Admin but can't link other Twitter accounts
- May use
- Everyone invited to TweetDeck Team has to use TweetDeck to post/use Twitter
12.2.6. Twitter Ads
- Objective
- Awarenes: promote Tweets and maximize reach
- Engagement: promote Tweets and get more Retweets, likes and replies
- Followers: promote account
- Website clicks and App installs
- Audience
- Geographic
- Followers of a notable account
- Human interests
- Bidding
- Auto bidding
- Pay for interaction: new follower, a click on your website
- Budget: no minimum, daily budget
- Creative
- Call-to-actions
- Twitter Ads Manager: ads.twitter.com
User management
- Access Levels
- Account administrator (AA)
- Can create AA and AM
- Ad Manager (AM)
- Can't create users
- Creative manager
- Campaign analyst
- Organic analyst: analytics.twitter.com
- Partner audience manager
12.3. Facebook
12.3.1. Facebook for developers
- https://developers.facebook.com/
- My Apps > New App
- Ok to select no scenario
- Later go to App Profile to specify
- Display Name
- App name
- (no term)
- App Description
- (no term)
- Contact Email
- (no term)
- App Icon
- (no term)
- Privacy Policy URL
- (no term)
- Business Use
- one platform type for an App e.g. can only have 1 Website platform for an App
- Website
- App Domains
- abc.com
- Site URL (full URL)
- https://www.abc.com
- Website
- Add Product
Facebook Login
- Platform
- Web
- Site URL
- https://www.abc.com
- (no term)
- Products > Facebook Login
- Valid OAuth redirect URIs
- https://www.abc.com/wp-admin/admin.php?page=nxssnap
12.3.2. Image Guidelines
- https://developers.facebook.com/docs/sharing/best-practices/#images
- Width > 1080, or minimum 600. 1:1 for ad creative
- Sharing Debugger">
- Link Preview with image
- # of likes, shares and comments
- html:og
12.3.3. Page
- Settings
- Page Roles
- Page Owner
- admins of a FB account can manage roles and other permissions on this Page
- Agencies
- assign approved permissions on this Page as a whole to all people inside an agency
- Existing Page Roles
- individuals
- Page Roles
12.4. Instagram
- Personal Profile
- Private account
- Linking to multiple FB Pages
- Business Profile account
- Create a personal/normal Insta account, login to Facebook which has permission to manage an FB Page
- On Insta,
Switch to Business Profileand select the FB Page
- On Insta,
- Access to Instagram Analytics
- Promote posts as ads
- Promote posts e.g. boost a post on FB Page
- Contact Button near the top
- Shopping + Checkout
- require your approval to allow other Insta accounts to tag you as a branded content partner
- Add links to Instagram Stories
- Schedule and auto publish posts to Insta
- DM quick replies
- Can't go private on Instagram
- Create a personal/normal Insta account, login to Facebook which has permission to manage an FB Page
- Creator Profile
- For influencers
- Direct Messages filtering
- Access creator-specific analytics including follow/unfollow metric and engagement stats
- Shopping + Checkout
- Photos and Videos Requirements
- Limited by Instagram Graph API which 3rd party services use to auto publish
- https://developers.facebook.com/docs/instagram-api/reference/user/media#create-photo-container
- Photos
- Max file size
- 8MB
- Aspect ratio
- within 4:5 to 1.91:1
- Min resolution
- 150x150 (lower resolution will be scaled up to the minimum)
- Max resolution
- 1920x1080 (will be scaled down)
- JPEG only
12.5. YouTube
- Link to subscribe
https://www.youtube.com/channel/YOURHASEDCHANNELID?sub_confirmation=1- Recommended upload encoding settings
- https://support.google.com/youtube/answer/1722171?hl=en&ref_topic=2888648 youtube:encoding
12.6. LinkedIn
12.6.1. LinkedIn Page
- Add an admin
- An existing page admin has to have first degree connection with the target person who is going to be added as admin
12.6.2. LinkedIn Marketing Solutions
- User Role
- Account Manager
- manage users
- (no term)
- Campaign Manager
- (no term)
- Creative Manager
- (no term)
- Viewer
- (no term)
- Billing Admin
- Each account needs at least one billing admin
- Account creator is assigned as the billing admin
- LinkedIn advertising hierarchy
- account > campaign group > campaign > ad
- account
- currency, LinkedIn Company Page
- campaign group
- budget, start/end dates, status:active
- campaign
- define possible ad formats: Sponsored Content, Sponsored InMail and Text Ads
- account > campaign group > campaign > ad
- LinkedIn Insight Tag
- A LinkedIn account can have multiple accounts and each account associates with one company page. Each Insight tag is unique to one account and thus one company page
- An account can add multiple users. Each user under an account has a unique Insight tag
- A tag can be on multiple websites and domains have to be verified on LinkedIn Campaign Manager
- A tag can provide conversion tracking, learn audience and collect audience for remarketing
12.7. Buffer.com
-
Account Social Accounts Team Members Scheduled posts per social Free 3 0 10 Business: Medium 50 10 2000 - Free
- 3 social accounts, 0 team member, scheduled posts per social
- For each social account, define post time
13. SEO & Marketing
13.1. Starter Guide
13.2. Google Test Mobile Speed
13.3. Google My Business
- Alternative
- Bing places for business
Local directory management services
- Yellow Pages
- Superpages.com
- Yelp
- BBB.org
- Manta.com
- Yahoo yext.com
- AngiesList.com
- advicelocal.com
- synup.com
- brightlocal.com
- Data Aggregators
- InfoGroup.com
- Acxiom.com
- Insights
- Direct Search and Discovery Search
- Search views and Maps views
- Visit your website, Request directions, Call you, View photos
- Local Business Categories
- https://developers.google.com/my-business/
13.4. Google Analytics google:ga
13.4.1. Limits, Versions, Accounts
- GA sets and reads cookies for each unique domain e.g. www.example.com and dogtoys.example.com have 2 different cookies. Javascript
document.domain - GA for FireBase is free and limited to track mobile apps
- https://developers.google.com/analytics/devguides/collection/analyticsjs/limits-quotas
- GA 360 is a paid per year service that includes a Service Level Agreement (SLA) that supports higher hit volumes
- https://www.lunametrics.com/services/google-analytics-360-hub/feature-comparison/
- 10M hits per month free and up to several billions. Data freshness is 24-48 hours vs 4 hour guarantee (most 15-20min)
- Sampling
- Standard report always has unsampled data and if you add a segment, create a report, filter the report, then sampled data will be used
- sampling occurs in Property. Default sample is 250K visits (sessions) and can be changed to 500K per property
- sampling occurs in View. Default sample is 250k visits (sessions) and can be changed to 100M sessions per view
- View Filter won't affect sampling
- 360 can
- use Google BigQuery
- custom tables, unsampled reports
- Roll-up reporting aggregates data from multiple Analytics properties and lets you see that data together in the same reports. e.g. website and mobile app stats
- google:ga:limit:dimensions metrics
- GA
- 20 and 20. Value of custom dimension has max length 150 bytes around 150 characters
- 360
- 200 and 200. 10 times more custom dimensions & metrics
- Integration
- GA
- AdWords and AdSense. Import stats from AdWords to GA and GA exports remarketing audiences to AdWords
- 360
- GA + DoubleClick Campaign Manager, DoubleClick Bid Manager, DoubleClick for Publishers, Salesforce Marketing Cloud
- Custom Data Sources
- GA
- import
- 360
- query-time import custom data sources (compare historic data and view the new imported data)
- Standard Attribution Modeling Tool
- GA
- give some formula-based models for attributing conversions to channels (like first-touch, last-touch, time decay, etc.)
- 360
- But which of those is the “right” model? GA360 includes a data driven attribution model, which actually uses an algorithm on your data to understand where different channels make the most impact
- Funnel Reporting Options
- 360
- Enhanced custom funnel reporting options. e.g. How many users view the page then scroll to 50% then scroll to the end of article then scroll to the end of page
- An Analytics account (analytics account ID) can have up to 50 properties and each property can have up to 25 views
13.4.1.1. Hit limits
- 10 million hits per month per property
- applies to Web Property / Property / Tracking ID (e.g. UA-xxx)
- if go over
- you will be contacted asking you to upgrade to 360 or implement client sampling to reduce the amount of data being sent to Google Analytics
- GA 360
- up to several billions
- 200,000 hits per user per day and 500 hits per session (universal analytics)
- gtag.js, analytics.js, Android SDK, iOS SDK and Measurement Protocol
- 500 hits per session
- for legacy tracking library (ga.js)
- Session timeout
- default 30 minutes. Min 1 minute and Max 4 hours
- (no term)
- Tracker objects
- Start with 20 hits that are replenished at a rate of 2 hits per second
- gtag.js and analytics.js tracker object. Applies to all hits except for ecommerce (item or transaction)
- Start with 10 hits that are replenished at a rate of 1 hit per second
- ga.js
Cut down hits per month
- Separate traffic by setting up new property
- Cut down spam traffic
13.4.2. Views, Filters, Segments
- Admin > Account > Property > Views
- Up to 25 views per property. 35 days to recover a deleted View
- (no term)
- Views collect data when they are created
- (no term)
- Views may have View Filters and changes on Views and Filters will affect future data only
- (no term)
- Under Views, there're a lot of reports (Standard and Custom). Apply Segments to reports to filter out data without destroying Views data
- (no term)
- Always have a View without Filters to capture all data
- 3 views
- Unfiltered, Main and Test. Test new filters on Test View first and then apply to actual views
13.4.2.1. View Filter
A Filter is to filter out the types of traffic you don't want and save the data to your View.
- Filter might take 24 hours to activate
- Filter will limit the stats data from the moment it is fully activated
- Filters are account-level. If filters are changed on View level, they are also changed across all properties of that account
In order to show only traffic that are on a set of pages, you can:
- Figure out a Regex statement and use it as the filter that is inside the View > Default Segment ○ You can't save this filter. You have to do it manually every time you want the report ○ You can see the filtered results right away and you can see previous stats
- Use JavaScript to submit Custom Variable on those pages and set to Page-level. ○ Create a new View with Admin > Property > View > Filter ○ You have to wait 24 hours to let the stats to be recorded from then ○ You can't see the previous stats
You can refer to this for constructing your ga:regex in Admin > Property > View > Filter or the normal filter.
Custom Filter - Advanced
- Extract 2 values from 2 fields (Field A and B), and output that to another field
- The output field can be one of these
- Custom Field
- System field (e.g.
Request URI, override the field value) - User Defined value
Request URIincludes URL parameters but without domain and protocol- Custom Field 1 and 2 are used only in filters. Appears on GA report Audience > Custom > Custom Variables
- User Defined value can be set by JavaScript and it's a single value. Appears on GA report Audience > Custom > User Defined
- Example, only include traffic that URI has
utm_source=MV_*andutm_medium=email
Advanced Filter
- Field A
(utm_source=MV_[^&]*)- Field B
(utm_medium=email)- Output to Custom Field 1
$A1&$B1- (no term)
- Setting Field 1 and 2 are required, Override Output Field
Custom Filter
- Include, Custom Field 1,
utm_source=[^&]*&utm_medium=email
13.4.2.2. System Filters
13.4.2.3. Include domain names (Custom Filter)
- Filter Type
- Custom filter > Advanced
- Field A
- Hostname Extract A:
(.*) - Field B
- Request URI Extract:
(.*) - Output To
- Request URI Constructor:
$A1$B1
13.4.3. Dimensions vs Metrics ga:dimensions ga:metrics ga:scope
- Each cell shows the value in the corresponding metric column that also belong to the corresponding dimension row
- e.g. metric is Goal completion and dimension is country. Table shows number of goal completions in each country
- Which dimension value has the highest metric? e.g. Which page category (custom dimension) has the highest pageviews (built-in metrics)
- Filters are only applied to dimensions
- Dimensions are fields that can be grouped by in SQL
- https://support.google.com/analytics/answer/2709828#scope
- Product
- value is applied to the product for which it has been set (Enhanced Ecommerce only)
- Hit
- Categorize actions e.g. pageview, event. A hit can associate with multiple products
- Session
- Categorize sessions, each session may have multiple hits. Timeout limit by default is 30 minutes. Max can go to 4 hours. Admin > Property
- User
- Categorize users. A cookie that has sessions
- the true total of each metric might be larger than the top of the total shown in the top row. Percentage of shown in each cell is calculated against the true total of each metric/column. Each column in the top row total, I think, is the unique metric
- e.g. Primary:Browser (user) and Secondary:Post Terms (hit)
- Last hit's dimension send defines the hit, session and level dimension
- Metric can only have Hit or Product level
- Metrics can be set as KPI's to indicate macro or micro conversions
- sum, ratio
- Metrics are usually submitted using Event Hits and may associate with custom dimension
- An example of a custom metric may be an Event Value of a certain Event
- All standard Dimensions & Metrics Explorer in what scope
13.4.3.1. Custom dimensions and metrics ga:custom dimensions
- https://support.google.com/analytics/answer/2709828
- Custom Dimension can be used as secondary dimension in standard reports but as primary in Custom Reports and can be used as ga:segment
- Allow you to combine Analytics data with non-Analytics data, e.g. CRM data. For example:
- If you store the gender of signed-in users in a CRM system, you could combine this information with your Analytics data to see Pageviews by gender
- If you're a game developer, metrics like "level completions" or "high score" may be more relevant to you than pre-defined metrics like Screenviews. By tracking this data with custom metrics, you can track progress against your most important metrics in flexible and easy-to-read custom reports
- Custom dimensions can appear as primary dimensions in Custom Reports. You can also use them as Segments and secondary dimensions in standard reports.
- Up to 20 custom dimensions and 20 custom metrics
13.4.3.2. Calculated Metric
Admin > Property > View > Calculated Metrics
13.4.4. Bounce rate
- A bounce is counted when there's only one pageview in one user session
- A bounce is calculated specifically as a session that triggers only a single request to the Analytics server, such as when a user opens a single page on your site and then exits without triggering any other requests to the Analytics server during that session
- Bounce rate is single-page sessions divided by all sessions, or the percentage of all sessions on your site in which users viewed only a single page and triggered only a single request to the Analytics server
- These single-page sessions have a session duration of 0 seconds since there are no subsequent hits after the first one that would let Analytics calculate the length of the session
13.4.5. Audience
13.4.5.1. User, New User, Session ga:identifiers
- Refer to ga:gtag:config
- UA-12315-12 (tid) where 12315 is Analytics account ID or Analytics account number
- UA-12315-12 is the same as the tracking id
- The total number of users for the requested time period. Long time cookie. Same browser and same device
- The number of users whose session was marked as a first-time session
- Short time cookie. Each session from a unique user will get its own incremental index starting from 1 for the first session. Subsequent sessions do not change previous session indices. For example, if a user has 4 sessions to the website, sessionCount for that user will have 4 distinct values of ‘1’ through ‘4’.
| Cookie | Expiration | Description |
| _ga | 2 yrs | Client ID |
| _gid | 24 hours | Used to distinguish users |
| _gat | 1 minute | |
| AMP_TOKEN | 30 secs to 1 yr | Client ID from AMP Client ID service. |
| _gac_<property-id> | 90 days | GA is linked with AdWords account, AdWords website conversion tags |
_gaor_gidcookie example- Value
GA1.2.2033959936.1513880474- Client ID
2033959936.1513880474
- Domain
- e.g.
.myweb.ca
13.4.5.2. Active user
Users that initiate session(s) in the last n days. Also called site reach, stickiness. 7-Day Active Users: the number of unique users who initiated sessions on your site or app from January 22 through January 28 (the last 7 days of your date range).
13.4.5.3. Cohort Analysis
Cohort analysis helps you understand the behavior of component groups of users apart from your user population as a whole. Examples of how you can use cohort analysis include to see how the behavior and performance of individual.
e.g. Group an audience based on their acquisition date and then compare behavior metrics over a series of weeks
Cohort type :: the only option now is Acquisition Date :: the first time a user is recognized as interacting with your content. When selected in the Cohort Analysis report, the cohorts are grouped based on when users started their first sessions.
Say metric is Pageviews and Date Range is 7 days. Today's April 12 For users visited the website for the first time on April 12-7 = April 5, they also visit n PageViews on Day 1 the next day April 6.
Metric User Retention :: The number of users in the cohort who returned in the Nth time period (day, week, month) divided by the total number of users in the cohort.
e.g. 3.02% of the users who visited the website for the first time on April 5, also visit the website on the next day, April 6 (Day 1).
13.4.5.4. Audiences
- This report shows only audiences you create in Analytics. The new audience will have up to 30 days of data and you can use it in 24-48 hours
- At any one time, you can have a maximum of 20 audiences published to Analytics. 2000 audiences per property
- Audiences are available only in the view in which you create them
- Audience can be published to only 1 destination of each type e.g. Google Ads or Display & Video 360, Optimize and Analytics. Once an audience is published to Google Ads/Google Marketing Platform, the destination cannot be changed
- Audience includes Age, Gender or nay of the Interest dimensions, audience can only be published to Google Ads (Display) and Analytics
- Audience includes Sequences, it cannot be published to Analytics
- Search and Display audiences are backfilled with up to 30 days of data
- Audiences that are based on Session Date dimension need 5 days before the session date
- Audiences that are based on custom dimensions that use query-time import mode are not supported. Audience data is evaluated at processing time
- Create Audience ga:audience:create
- Enable Demographics and Intersts Report under Property > Property Settings > Advertising Features ga:audience:interests
- https://support.google.com/analytics/answer/2611404
- An Analytics account can have 2000 (Remarketing) Audiences
- Go to Admin > a property > Audience Definitions > Audiences > New Audience
- Create an audience is like ga:segment:create
- https://support.google.com/analytics/answer/6015314
- Refer to ga:remarketing
- Remarketing audiences examples
- https://support.google.com/analytics/answer/2611820
- State-based audiences examples
- https://support.google.com/analytics/answer/6212382
- AdWords Search Remarketing
- requires 1000+ unique users (cookies)
- GA's demographics dimensions cannot be used for AdWords Search Remarketing
- AdWords Display Remarketing
- requires 100+ unique users
- https://support.google.com/analytics/answer/2611268
- Preconfig new audience
- Smart List: Let Google manage the audience for you
- All users to your site or app who already have the necessary advertising cookies or mobile-advertising IDs
- Any users who have conducted only one session on your site or app
- Any users who have conducted more than one session on your site or app
- Users who visited a specific section of my site/app
- Click the edit icon, and enter the URL of a page or directory on your site, or a screen in your app. This option uses the contains match type, and matches any URL that contains the string you enter here.
- If there are more than 1000 page/screen URLs for your site/app, then Analytics displays matches as you enter text only if matches are found within the first 1000 URLs. If there are no matches in the first 1000 URLs, then Analytics displays nothing. In this case, you can copy and paste the URL from a browser, or from some other source of URLs like a spreadsheet
- Click the edit icon, and select a goal from the menu. This option requires that you have previously configured Analytics Goals
- This is already configured to include any user with more than zero transactions
- Use Audiences
- https://support.google.com/analytics/answer/7280979
- After you create an audience and publish it to Analytics, it is available as a primary dimension in the Audience > Audiences report, as a secondary dimension in other reports, and as a dimension in segments, custom reports, and custom funnels.
13.4.5.5. Demographics ga:audience:demographics
- Enable Demographics and Interests Report
- https://support.google.com/analytics/answer/2819948
- Enable Advertising Feautures (Advertising Reporting Features)
- Admin > Property > Tracking Info > Data Collection
13.4.5.6. Interests ga:audience:interests
- Enable Demographics and Interests Report
- https://support.google.com/analytics/answer/2819948
- Affinity
- with passion or interests in an area but not necessarily looking to buy
- In-Market Segments
- looking to buy
- (no term)
- Other
13.4.5.7. User Explorer
Instead of aggregating users or user sessions, this shows each user/session's
- Sessions
Total Session Duration / Sessions. The last session that a user stops navigating to another page/hit has 0 second duration. High bounce rate may cause this metric to be low- Bounce rate
- Revenue
- Transactions
- Goal Conversion Rate
13.4.5.8. Benchmarking
You can choose from over 1600 industry categories, using a menu in the Benchmarking reports. You can further refine the data by geographic location and select from seven traffic size classifications, allowing you to compare your property against properties with similar traffic levels in your industry. For example, you can compare your property with all properties in the “All Hotels and Accommodations” industry in the United Kingdom that receive 500 to 1000 average daily sessions.
Compare in
- Channels
- Location
- Devices
13.4.5.9. Users Flow
Segment does not work.. To show google/cpc, change the first dimension item from Country to Source/Medium = google/cpc
13.4.6. Acquisition ga:channel
- Channel is the traffic source
- Change the Default Channel Grouping
- Admin > Property > View > Channel Settings > Channel Grouping > Default Channel Grouping
- Direct
- Search (Organic + Paid)
- Referral
- Social
- Organic Search
- Paid Search
- Other Advertising
- Display
- Add a custom channel grouping
- Admin > Property > View > Custom Channel Grouping
- e.g. define a channel to include traffic from a campaign containing the term January (e.g., January1, 2ndJanuary, January). Instead of specifying the rule as January (which will only return January), enter it as
.*January.*
- e.g. define a channel to include traffic from a campaign containing the term January (e.g., January1, 2ndJanuary, January). Instead of specifying the rule as January (which will only return January), enter it as
- is more like channels, and source provides more specific about the medium
- Every referral to a website also has a medium. Possible medium include:
- organic
- unpaid search
- cpc
- cost per click, i.e. paid search
- (no term)
- referral
- the name of a custom medium you have created)
- none
- direct traffic has a medium of “none”
- Every referral to a website also has a medium. Possible medium include:
- Every referral to a web site has an origin, or source. Possible sources include:
- the name of a search engine
- facebook.com
- the name of a referring site
- spring_newsletter
- the name of one of your newsletters
- direct
- users that typed your URL directly into their browser, or who had bookmarked your site
- When SSL search is employed, Keyword will have the value (not provided)
- the name of the referring AdWords campaign or a custom campaign that you have created
- a specific link or content item in a custom campaign. For example, if you have two call-to-action links within the same email message, you can use different Content values to differentiate them so that you can tell which version is most effective. AdWords Ad or extension name
- search keywords
- is case sensitive and spaces are converted to
_ - Refer to google:campaign-url-builder
| Email campaign | Paid search campaign | |
| Campaign Source | newsletter1 | yahoo |
| Campaign Medium | cpc | |
| Campaign Term | the search term associated with this traffic | |
| Campaign Content | call_to_action_2 | |
| Campaign Name | productxyz | productxyz |
- Use
source/mediumasgoogle/cpcto have AdWords Search and Display Networks - To distinguish Paid Search and Display, use Default Channel Grouping
- To distinguish Google Search and Search Partners, use Ad Distribution Network
- Once AdWords account(s) is linked to GA, you can see relevant reports in Acquisition > AdWords
- Accounts, Campaigns, Treemaps, Bid Adjustment, Keywords, Search Queries, Hour of Day, Destination URLs, Display Targeting, Video Campaigns, Shopping Campaigns
- Acquisition
- Clicks, Cost, CPC, Users
- Behavior
- Bounce Rate, Pages/Session
- Conversion by each goal
- Conversion rate, Transactions, Revenue
For some dimension, you can filter by Device type on the top.
13.4.7. Behavior
- How users navigate between pages
- Speed, Search, Events
13.4.7.1. Average Time on Page ga:avg. time on page
- The average length of time users spend viewing a page/specific set of pages
- Because the last pageview of a user does not have a next pageview, so Google cannot calculate the last page duration. That's why the formula minus the Exit pageviews
- It is usually higher than ga:avg. session duration because Exit pageviews is not counted in this metric. If it's lower, it might mean each session has a lot of pageviews
Time on Page / (Pageviews - Exits)
13.4.7.2. Content Grouping
- Standard Report
- Admin > Property > View > Content Grouping
- (no term)
- Content Grouping is not suited for grouping WordPress post categories as a post can have multiple categories and the same Content Grouping can have only one value
- Methods to create a Content Group
- in order. First matching method defines the content group
- Modify the tracking code on each page
- Extract pages with regex (Page URL, Page Title)
- Create rules to include pages in a group (Page URL, Page Title, Screen name)
- (no term)
- Usually Content Grouping is submitted with pageview hit type
Behavior > Site Content > all reports
- Content Group has to be set before or at the moment pageview hit is sent
<script async src="https://www.googletagmanager.com/gtag/js?id=GA_TRACKING_ID"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); // original :: config and send a pageview //gtag('config', 'GA_TRACKING_ID'); // change to gtag('config', 'GA_TRACKING_ID', {'content_group1': 'shoes'}); // or // gtag('set', {'content_group1': ''}); // gtag('config', 'GA_TRACKING_ID'); </script>
<script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-XXXXX-Y', 'auto'); // set before pageview is sent ga('set', 'contentGroup1', 'my_group_name'); ga('send', 'pageview'); </script>
- It can be sent in other event types e.g. hit as well
- Haven't tried in code but tried in GTM
- Standard report Behavior > Events > Pages, group events by Content Group
- These are not in the Site Content reports
- These do not change the previous pageview hit Content Group
- (no term)
- Content Grouping is not shown on Realtime reports
- Also used in Segment's Conditions Filter
- content group has to be set before or at the moment pageview hit is sent
- Page Group 1
- Landing Page Group 1
- Next Page Group 1
- (no term)
- e.g. Webpages have
- Content Grouping:Clothing at index 1
- a value belongs to Content Grouping:Clothing, also a Content Grouping:Men at index 2
- Shirts
- a value belongs to Content Grouping:Men
- Pants
- a value belongs to Content Grouping:Men
- (no term)
- Accessories
- a value belongs to Content Grouping:Clothing, also a Content Grouping:Women at index 3
- Tops
- a value belongs to Content Grouping:Women
- (no term)
- Slacks
- (no term)
- Skirts & Dresses
- (no term)
- Accessories
- Clothing, Men, and Women are options in the Primary Dimension > Content Grouping menu
- When you select Clothing as the primary dimension, Men and Women are the dimension values in the first column of the report table
- When you select Men as the primary dimension, Shirts, Pants, and Accessories are the dimension values in the first column of the table
- For each dimension value (Content Group), you see behavioral metrics like Pageviews and Bounce Rate
13.4.7.3. Behavior Flow
Change the dimension (green box on the left) to Campaign to show AdWords campaign(s) behavior flow.
13.4.8. Conversions
Goals, Ecommerce, Multi-Channel Funnels, Attribution
- Multi-Channel Funnels
Assisted Conversions What are the sources contribution in terms of:
- Last interaction
- the interaction that immediately precedes the conversion.
- Assist interaction
- any interaction that is on the conversion path but is not the last interaction.
- First interaction
- the first interaction on the conversion path; it's a kind of assist interaction.
- Time Lag
- how many days a conversion is made
- Top Conversion Paths
- what sources were went through before a conversion is made
- Path length
- how many interactions took before a conversion is made.
13.4.9. Add Google Search Console to a property
There will be a new section in the report called Acquisition > Search Console Need to go back to Search Console to see keywords report GA shows keyword only for non-google-logged-in users that use google to search google:ga:search console
13.4.10. Campaigns and Goals track across user session
13.4.11. Goal vs Event
- A goal provides additional data such as conversion rate in Acquisition reports
- A goal will only record once per session. Newsletter sign up twice with 2 emails, only one conversion is recorded
- An event can be anything. A goal has to be a destination, duration, pages per session or based on an event
- A goal can be based on an event
- Goal flow report
- Goal cannot be shared across Views nor Properties. It has to be created in each View unless when View is duplicated
- Goal ID 1 to 5 is Goal Set 1, ID 6 to 10 is Goal Set 2 etc.
Types of a Goal
- Desitnation
- the only type that is possible to track funnels
- Duration
- Sessions that last a specific amount of time or longer
- (no term)
- Pages/Screens per session
- Event
- multiple events in one session but event goal is counted as one
- A conversion is counted once per session per goal
- When Event value is empty on Goal setup page, it means it could be any value
- (no term)
- Smart Goal
- (no term)
- Destination Goal match types: begin with, equal to & RegEx
Events are good for
- Downloaded file (shows which pages have the pdf file link)
- Link (outbound)
- The first 10 event hits sent to Analytics are tracked immediately, thereafter tracking is rate limited to one event hit per second
- It's better to setup events and setup goals as the event goal type
13.4.12. Segment ga:segment
- A Segment is to filter out the
user sessionsand furtheruserthat you don't want without destroying your View data - Segment occurs after sampling in Property for GA Standard
- Segment by default is visible in any View across account and can only be edited by the creator. Options
- Creator can apply/edit Segment in any View
- Creator can apply/edit Segment
- Collaborators and Creator can apply/edit Segment in this View
- Admin > Property > View > Segment > Advanced > Conditions Filter
- can use regex
- All Users and New Users. Metrics inside each type of report will be divided into 2 so that you can compare
- Say you want to use Segment to only include user sessions that have been to a url
- Go to the newly created Segment, you will still see those user sessions have been to other pages
- https://searchenginewatch.com/sew/how-to/2268458/16-secret-google-analytics-advanced-segments-worth-their-weight-in-gold
13.4.12.1. Converter segment ga:segment:converter
- Refer to ga:goal
- Include user sessions that have finished a goal. You can copy this system goal and make it your own e.g. finish a specific goal
- Some stats you may find is that people who convert have lower bounce rate, higher pages/session, higher avg. session duration.
13.4.12.2. Custom segment - AdWords Search adwords:segment:search
Users Include Source/Medium = google/cpc AND Default Channel Grouping = Paid Search
13.4.12.3. Create a segment ga:segment:create
- ga:audience:create may use a segment
- https://support.google.com/analytics/answer/3124493
13.4.13. Tracking Code - gtag.js, analytics.js ga:gtag
13.4.13.1. Basics
- The most recent version is gtag.js (Global site tag)
- Google Analytics gtags.js Documentation
- https://developers.google.com/gtagjs/
- gtag.js API
- Migrate from analytics.js to gtag.js
- Google Analytics, Campaign Manager, Google Ads and Google Marketing Platform (e.g. Display & Video 360, Search Ads 360) use the same gtag.js
- Other version
- analytics.js (Universal) ga:ga
#+NAME gtag.js
<!-- Global site tag (gtag.js) - Google Analytics --> <script async src="https://www.googletagmanager.com/gtag/js?id=UA-xxx"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-XXX'); </script>
#+NAME analytics.js
<script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-XXXXX-Y', 'auto'); ga('send', 'pageview'); </script>
Old ga.js
<script type="text/javascript"> // IMPORTANT: Remove this code snippet when upgrading to analytics.js var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-XXXX-Y']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script>
13.4.13.2. Custom parameters for dynamic remarketing ga:dynamic remarketing
https://support.google.com/adwords/answer/3103357
<!-- Global Site Tag (gtag.js) - Google AdWords: 123456789 --> <script async src="https://www.googletagmanager.com/gtag/js?id=AW-123456789"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'AW-123456789'); </script> <!-- Event snippet for Example dynamic remarketing page --> <script> gtag('event', 'page_view', { 'send_to': 'AW-123456789', 'ecomm_prodid': 'REPLACE_WITH_STRING_VALUE', 'ecomm_pagetype': 'REPLACE_WITH_STRING_VALUE', 'ecomm_totalvalue': 'REPLACE_WITH_STRING_VALUE' }); </script>
Have to use certain parameters for certain business type. Here're the list of parameters.
First, setup Custom Dimensions based on business type. https://support.google.com/analytics/answer/3455600
Next, implement tags like the above on the website
Then setup these Dynamic Attributes in GA > Admin > Property > Audience Definitions > Dynamic Attributes. This step also shares the attributes with AdWords accounts. https://support.google.com/analytics/answer/6002231
Then you can setup Dynamic Remarketing on AdWords
Maximum duration that a user can be included in a remarketing audience is 540 days.
13.4.13.3. gtag.js config analytics.js create ga:gtag:config ga:ga:create
- ga:gtag:config:groups
Refer to ga:gtag:event:send_to
// Default config sends a pageview hit. Don't send a pageview gtag('config', 'UA-xxx', { 'send_page_view': false }); // Send a virtual pageview gtag('config', 'GA_MEASUREMENT_ID', {'page_path': '/new-page.html'});
- Cookie domain
- Most cases set it to 'auto', it sets the
_gacookie to the highest level domain it can. e.g. if your website address isblog.example.co.uk, gtag.js will set the cookie domain toexample.co.uk. If it detects that you're running a server locally (e.g. localhost), it automatically sets the cookie_domain tonone - GA requests on websites that are subdomains and ancestor subdomains can share the same cookie
- If
cookie_domainis not set, it will be set to the current domain Turn off automatic cookie domain configuration, update the config for your property to specify a value for the
cookie_domainparametergtag('config', 'GA_TRACKING_ID', { 'cookie_domain': 'blog.example.co.uk' }); ga('create', 'GA_TRACKING_ID', 'auto'); // or ga('create', 'GA_TRACKING_ID', {'cookieDomain' : 'auto'});
- Most cases set it to 'auto', it sets the
- (no term)
Cross-domain tracking allows you to see sessions from 2 sites as single session. Also called site linking. This code will append the linker parameter to any links on the page that point to the target domain 'example.com':
gtag('config', 'GA_TRACKING_ID', { 'linker': { 'domains': ['example.com'] } });
- If the destination domain has been configured to automatically link domains, it will accept linker parameters by default
If the destination domain is not configured to automatically link domains, you can instruct the destination page to look for linker parameters by setting the accept_incoming property of the linker parameter to true on the destination property's config:
gtag('config', 'GA_TRACKING_ID', { 'linker': { 'accept_incoming': true } });
Single snippet on all domains
// on example-1.com gtag('config', 'GA_TRACKING_ID_1', { 'linker': { 'domains': ['example-1.com', 'example-2.com'] } }); // on example-2.com gtag('config', 'GA_TRACKING_ID_2', { 'linker': { 'domains': ['example-1.com', 'example-2.com'] } });
13.4.13.4. gtag.js set ga:gtag:set
Set parameters that will be associated with every subsequent event on the page
gtag('set', { 'country': 'US', 'currency': 'USD', 'metric1', 1 });
13.4.13.5. gtag.js event ga:gtag:event
13.4.13.6. Send to multiple property ids ga:gtag:event:send_to
ga('create', 'UA-XXXXX-Y', 'auto'); ga('create', 'UA-XXXXX-Z', 'auto', 'clientTracker'); ga('send', 'pageview'); ga('clientTracker.send', 'pageview');
gtag.js
gtag('config', 'GA-TRACKING_ID-1', { 'groups': 'group1' }); gtag('config', 'GA-TRACKING_ID-2', { 'groups': 'group1' }); // Routes to 'GA-TRACKING_ID-1' and 'GA-TRACKING_ID-2' gtag('event', 'sign_in', { 'send_to': 'group1' }); // The following two lines are equivalent: gtag('config', 'GA-TRACKING_ID-1'); gtag('config', 'GA-TRACKING_ID-1', { 'groups': 'default' });
- Full example
- send to a specific property instead of groups
<script async src="https://www.googletagmanager.com/gtag/js?id=GA-TRACKING_ID-1"> </script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); // Global configs gtag('config', 'GA-TRACKING_ID-1'); gtag('config', 'AW-CONVERSION_ID'); gtag('config', 'DC-FLOODLIGHT_ID'); // Track AdWords conversions gtag('event', 'conversion', { 'send_to': 'AW-CONVERSION_ID/AbC-D_efG-h12_34-567', 'value': 1.0, 'currency': 'USD' }); // Track Floodlight conversions gtag('event', 'conversion', { 'allow_custom_scripts': true, 'send_to': 'DC-FLOODLIGHT_ID/actions/locat304+standard' }); // route ecommerce add_to_cart event to AdWords and Analytics gtag('event', 'add_to_cart', { 'send_to': [ 'GA-TRACKING_ID-1', 'AW-CONVERSION_ID' ], 'items': [ 'id': 'U1234', 'ecomm_prodid': 'U1234', 'name': 'Argyle Funky Winklepickers', 'list': 'Search Results', 'category': 'Footwear', 'quantity': 1, 'ecomm_totalvalue': 123.45, 'price': 123.45 ] }); </script>
13.4.13.7. ga Command Queue and Object Methods
https://developers.google.com/analytics/devguides/collection/analyticsjs/tracking-snippet-reference
<!-- Google Analytics --> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-XXXXX-Y', 'auto'); // create a tracker instance ga('send', 'pageview'); </script> <!-- End Google Analytics -->
ga('command_name', ...); // this is called command queue. Command can be a tracker method, a ga Object Method, or a method that is defined in a plugin // Use a ready callback to call a ga Object Method // ga.getAll() // wrong ga(function(tracker) { var trackers = ga.getAll(); });
13.4.13.8. ga Command Queue
Add a command to the queue :: ga(command, […fields], [fieldsObject])
- command
- [trackerName.][pluginName:]methodName
- The name of the method to be scheduled for execution. When not specifying a plugin name, this method must be one of the command methods listed below.
- …fields
- One or more optional convenience parameters for quickly specifying common fields.
- fieldsObject
- An object for specifying any remaining values not specified in any of the fields parameters.
Commands
- create
- create a tracker instance. same as ga Object Method create
- remove
- remove a tracker instance. same as ga Object Method remove
- send
- same as ga:tracker:send ga:ga:send
- set
- set a field on a tracker object. Same as ga:tracker:set
- require
- require an analytics.js plugin
- provide
- provide an analytics.js plugin and its methods for use with the ga() command queue. Add command
13.4.13.9. ga Object Method
create :: create a tracker instance getByName :: get a tracker instance by name getAll :: get all tracker instances remove :: remove a tracker instance
13.4.14. Tracker Object Methods
- get
- get the value of a field stored on the tracker ga:tracker:get
- set
- set a field value on the tracker ga:tracker:set
- send
- send a hit to Google Analytics. Same as ga:ga:send
13.4.14.1. set ga:ga:set
When a tracker's field is set, tracker will use that as default for all following actions/commands/methods e.g. send a hit. Use custom variable to subsitute into each action's required parameters instead of overwrite a tracker's field.
13.4.14.2. send ga:tracker:send
tracker.send([hitType], [...fields], [fieldsObject]);ga('[trackerName.]send', [hitType], [...fields], [fieldsObject]);- Used in ga:goal and funnel
- Use ga:fieldsObject
- hitType
- pageview
- ga:fieldsObject:page
- event
- Custom event
- social
- timing
- Send pageview - Page Tracking
- https://developers.google.com/analytics/devguides/collection/analyticsjs/pages
ga('send', 'pageview', [page], [fieldsObject]);
Pageview fields (tracker object fields)
- title
- It's set to document.title when tracker is created
- location
- either location or the page is required. It's set to document.location when tracker is created
- The anchor portion is ignored. Including campaign parameters in anchor but they're processed in a special way
- page
- default is not set. If set, location field gets overwritten when send method is run but tracker's location field is still not modified. ga:fieldsObject:page
Always define tracker.page which is only used in send method. Don't modify location of document.location. Or you can use self define page variable and pass it to send method without changing tracker.page which will always be used in send method.
For SPA, you can use a plugin to track complex pageviews :: https://github.com/googleanalytics/autotrack
ga('send', 'pageview'); ga('send', { hitType: 'pageview', page: location.pathname }); ga(function(tracker) { // Sets the page field to "/about.html". tracker.set('page', '/about.html'); }); // normalize /user/USER_ID/profile, /user/USER_ID/account, /user/USER_ID/notifications with thousands of USER_ID // to // /user/profile, /user/account, /user/notifications if (document.location.pathname.indexOf('user/' + userID) > -1) { var page = document.location.pathname.replace('user/' + userID, 'user'); ga('send', 'pageview', page); } ga(function(tracker) { // Sends an event hit for the tracker named "myTracker" with the // following category, action, and label, and sets the nonInteraction // field value to true. tracker.send('event', 'link', 'click', 'http://example.com', { nonInteraction: true }); });
- Funnel and Goal (send pageview) ga:goal and funnel
Goal is defined in Admin > Account > Property > View Funnel is step to a Goal.
Goal or funnel must use the pageview hitType.
When path is very dynamic, you may need to send pageview with a static path in order to capture goal or funnel destination URL.
And you may need to filter out those destination URL's in another GA View. Don't filter the current GA View.
View the Virtual Pageview in Reporting > Real-Time > Content View the converted goal in real time Reporting > Real-Time > Conversions
ga('send', 'pageview', '/ga_goal/goal1/goal1_variant1.html'); ga('send', 'pageview', '/ga_goal/goal1/goal1_variant2.html'); ga('send', 'pageview', '/ga_goal/goal1/step1/goal1_variant1.html'); ga('send', 'pageview', '/ga_goal/goal1/step2/goal1_variant1.html'); // ... gtag('config', 'GA_TRACKING_ID', { 'page_title' : 'homepage', 'page_path': '/ga_goal/goal1/goal1_variant1.html' });
- Send Custom Event - play, call, scroll ga:send event
- https://developers.google.com/analytics/devguides/collection/gtagjs/events
- https://support.google.com/analytics/answer/1033068#Anatomy
- Reporting > Behavior > Events
- Category > Action > Label > Value
- Category
- primary dimension
- Action
- secondary dimension
- Label
- maybe third dimension
- Syntax
- gtag.js
gtag('event', eventName_eventAction, eventParameters);- gtag.js
eventParameters
<eventName>- same as
eventAction <category>- the string that appears as the event category
<action>- the string that appears as the event action in Google Analytics Event reports
<label>- the string that appears as the event label
<value>- a non-negative integer that appears as the event value. Usually monetary value
non_interaction- By default, the event hit is considered an interaction hit, which means that it is included in bounce rate calculations. If the event is set to non_interaction true, then the event is not a hit
- gtag.js
- (no term)
- analytics.js
ga('send', 'event', [eventCategory], [eventAction], [eventLabel], [eventValue], [fieldsObject]);orga('send', 'event', {eventCategory: '', eventAction: '', eventlabel: '', eventValue: 123, 'dimension1': '', 'dimension2': '', 'metric1': 123 });orga('send', { hitType: 'event', eventCategory: '', eventAction: '', eventlabel: '', eventValue: 123, 'dimension1': '', 'dimension2': '', 'metric1': 123});
gtag('event', <action>, { 'event_category': <category>, 'event_label': <label>, 'value': <value> }); gtag('event', 'play', { 'event_category': 'Videos', 'event_label': 'Fall Campaign' }); $("a[href^='tel:+18881234567']").click(function(event){ gtag('event', 'click', { 'event_category': 'phone call', 'event_label': '1 (888) 123-4567', 'value': 100 }) }); $("a[href^='tel:']").click(function(event){ var $target=$(event.target); //console.log($target.attr('href')); gtag('event', 'click', { 'event_category': 'Call from website', 'event_label': $target.attr('href'), 'value': '100' }); }); // does not cover tel: and mailto: $('a[href]:not([href=""])').each(function (i) { if (typeof $(this).attr('href') !== 'undefined' && this.hostname && this.hostname.toLowerCase() !== location.hostname.toLowerCase()) { // outbound links $(this).on('click', function () { //console.log($(this).prop('href')); gtag('event', 'click', { 'event_category': 'Outbound-Link', 'event_label': $(this).prop('href'), 'value': '1', }) }) } }) gtag('event', 'video_auto_play_start', { 'event_label': 'My promotional video', 'event_category': 'video_auto_play', 'non_interaction': true });
// scroll depth compared to the root element var element_to_track = 'html'; var scroll_reach = 0; var temp_scroll_reach = 0; var scroll_reach_pixels = 0; var temp_scroll_reach_pixels = 0; $(window).scroll(function(){ temp_scroll_reach = Math.floor( 100 * ( $(window).scrollTop() + $(window).height() - $(element_to_track).position().top) / $(element_to_track).height() ); temp_scroll_reach_pixels = Math.floor($(window).scrollTop() + $(window).height()); if(temp_scroll_reach > scroll_reach){ scroll_reach = temp_scroll_reach; scroll_reach_pixels = temp_scroll_reach_pixels; } }); function sendGAreq() { // Old ga.js // _trackEvent(category, action, opt_label, opt_value, opt_noninteraction) //_gaq.push(['_trackEvent', 'Scroll Depth', 'Percentage', '', scroll_reach]); //_gaq.push(['_trackEvent', 'Scroll Depth', 'Pixels', '', scroll_reach_pixels]); // New analytics.js // ga('send', 'event', [eventCategory], [eventAction], [eventLabel], [eventValue], [fieldsObject]); ga('send', 'event', 'Scroll Depth', 'Percentage', '', scroll_reach); ga('send', 'event', 'Scroll Depth', 'Pixels', '', scroll_reach); } /* report only after a user exists the website */ $(window).bind('beforeunload', function() {sendGAreq();});
- Send dimension and metrics ga:gtag:event:custom dimension
- Configure and send custom dimensions
- Use chrome:google tag assistant to debug
- create Custom Report on GA
- Primary Dimension:Page, Secondary Dimension:Custom Dimension
- Metric: Users
- Wait a few minutes then create Segment using User contains Custom Dimension equal something and apply the segment to Behavior > All Pages
- Refer to
ga('create', 'UA-XXXX-Y', 'auto'); ga('send', 'pageview', { 'dimension2': 'My Custom Dimension' }); // or /* ga('create', 'UA-XXXX-Y', 'auto'); // Set value for custom dimension at index 1. ga('set', 'dimension2', 'dimensionValue'); // Send the custom dimension value with a pageview hit. ga('send', 'pageview'); */ // or after pageview hit is sent ga('send', 'event', 'any_eventCategory', 'any_eventAction', { 'dimension2': 'dimensionValue' }); // gtag has to run 2 commands to accomplish the same // Configures custom dimension<Index> to use the custom parameter // 'dimension_name' for 'GA_TRACKING_ID', where <Index> is a number // representing the index of the custom dimension. gtag('config', 'GA_TRACKING_ID', { 'custom_map': {'dimension<Index>': 'dimension_name'} }); // Sends the custom dimension to Google Analytics. gtag('event', 'any_event_name_eventAction', {'dimension_name': dimension_value}); // Maps 'dimension2' to 'age'. gtag('config', 'GA_TRACKING_ID', { 'custom_map': {'dimension2': 'age'} }); // Sends an event that passes 'age' as a parameter. gtag('event', 'age_dimension', {'age': 12}); // if multiple gtags are configured, send the event to the right UA-xxx // Refer to ga:gtag:event:send_to gtag('event', 'dimension_age', { 'send_to': 'UA-xxx', 'age': 12 });
- Configure and send custom metrics
// Configures custom metric<Index> to use the custom parameter // 'metric_name' for GA_TRACKING_ID, where <Index> is a number // representing the index of the custom metric. ga('send', 'event', 'category', 'action', { 'metric18': 8000 }); ga('send', 'event', 'category', 'action', { 'metric19': 24.99 }); // gtag has to run 2 commands to achieve the same gtag('config', 'GA_TRACKING_ID', { 'custom_map': {'metric<Index>': 'metric_name'} }); // Sends the custom dimension to Google Analytics. gtag('event', 'any_event_name', {'metric_name': metric_value}); // Maps 'metric5' to 'avg_page_load_time'. gtag('config', 'GA_TRACKING_ID', { 'custom_map': {'metric5': 'avg_page_load_time'} }); // Sends an event that passes 'avg_page_load_time' as a parameter. gtag('event', 'load_time_metric', {'avg_page_load_time': 1});
- Configure and send both custom dimensions and custom metrics
gtag('config', 'GA_TRACKING_ID', { 'custom_map': { 'dimension2': 'age', 'metric5': 'avg_page_load_time' } }); gtag('event', 'foo', {'age': 12, 'avg_page_load_time': 1});
// Set tracker's custom dimension so that all later hits will have this ga('set', 'dimension5', 'custom data'); // set both ga('set', { 'dimension5': 'custom dimension data', 'metric5': 'custom metric data' });
13.4.15. fieldsObject ga:fieldsObject
13.4.15.1. page ga:fieldsObject:page
text begins with /
13.4.15.2. Custom Dimensions and Metrics ga:fieldsObject:customs
13.4.15.3. hitCallback
var form = document.getElementById('signup-form'); // Adds a listener for the "submit" event. form.addEventListener('submit', function(event) { // Prevents the browser from submitting the form // and thus unloading the current page. event.preventDefault(); // Sends the event to Google Analytics and // resubmits the form once the hit is done. ga('send', 'event', 'Signup Form', 'submit', { hitCallback: function() { form.submit(); } }); // or ga('send', 'event', 'Signup Form', 'submit', { transport: 'beacon' }); form.submit(); });
13.4.16. Regular Expressions ga:regex
- https://support.google.com/analytics/answer/1034324?hl=en
. ? + * | ^ $ ( ) [ ] -- ^ \[0-9]{1,2}- http://www.mysite.com/shopping?qsrc=35&o=0&x=ref&stf=CA:LO&q=lava+lamp
1\.2\.3\.4|2\.2\.3\.4- http://www.analyticsmarket.com/freetools/regex-tester
- http://www.lunametrics.com/regex-book/Regular-Expressions-Google-Analytics.pdf
13.4.16.1. Include multiple words but order does not matter
word1+word2+word3
13.4.17. Analytics Solutions Gallery
- https://analytics.google.com/analytics/gallery
- A gallery may contain Custom Reports, Dashboard, Segments, etc.
- Imported gallery will show in Admin > View > Share Assets
- Dashboards, Custom Reports, Segments, Goals, and Custom Attribution Models can be shared in the Solutions Gallery
13.4.18. Email tracking, Measurement Protocol
- The Google Analytics Measurement Protocol allows developers to make HTTP requests to send raw user interaction data directly to Google Analytics servers
- https://developers.google.com/analytics/devguides/collection/protocol/v1/email
13.4.19. Site Search
- Admin > choose a View > View Settings
- Query Parameter
- up to 5 e.g.
s,qfora.ca/?s=abc&q=xyz- Strip query parameters out of URL
- only strip the parameters set up as above and not the other parameters
- Site search categories
- e.g.
a.ca/?s=abc&cat=xyzwherecatis the parameter to setup, which inidicates to search in what categories
ga('send', 'pageview', '/search_results.php?q=keyword')- Behavior > Site Search
- Usage
- Sessions that use Site Search vs not
- Search Terms
13.4.20. Custom Alerts
- Admin > choose a View > Custom Alerts
- Period
- Day, Week, Month
13.4.21. Google Analytis Core Reporting API
13.4.22. Remarketing with Analytics ga:remarketing
- https://support.google.com/analytics/topic/2611283
- Setup audience in GA can use all GA dimensions/metrics, sequence segment and regex compared to audience setup in AdWords with AdWords global site tag
- https://support.google.com/adwords/topic/3122879
- https://support.google.com/adwords/answer/6178664
- Admin > Property > Tracking Info > Data Collection
- Advertising Reporting Features
- GA collects ga:audience:demographics and ga:audience:interests data from
- 3rd party DoublClick cookie
- Android Advertising ID
- iOS Identifier for Advertisers (IDFA)
- enable to show demographics and interests on GA. These data are collected by DoubleClick and not GA
- Ensure Analytics > Admin > Account > Account Settings has enabled all Data Sharing or at least Google products & services
- https://support.google.com/adwords/answer/6209127
- Refer to google:ads:link GA
- One AdWords manager account can have multiple managed accounts. After linking, or those accounts can use the Google Analytics property
- ga:audience:create
- https://support.google.com/analytics/answer/3052703
13.4.23. Dynamic Remarketing with Analytics ga:dynamic remarketing
13.4.24. AMP ga:amp
13.4.25. Transfer a property
- Move Property
- https://support.google.com/analytics/answer/6370521
- (no term)
- Have
Manage UsersandEditpermissions on both the source and destination accounts (GA account level) - (no term)
- Property ID does not change
- (no term)
- Permissions
- Replace existing property and view permissions with permissions of the destination account
- the property and its views will inherit perms from the destination account
- Keep existing property and view permissions
- Source account perms will be copied to destination account. Users who have account-level access in the source account will have property-level access in the destination account
- (no term)
- Can't move a property if
The source account and the destination account are not in the same google:marketing platform organization
- It's easy to transfer a whole GA account (called product account) from one org to another on google:marketing platform
- Administration > Click on 3 dots on source org, move accounts from this org to another
Required permissions
- Source org: Org admin and Billing admin
- Destination org: Billing admin
- If the source org can't add you as an Org admin or Billing admin, then you should consider to unlink the GA account from source org and then link it to destination org after
- The service level for the property is set to 360 and the account link to the organization has not been verified
- The property is linked to Google Ad Manager (only GA 360 can link GAM)
- Unlink and relink after the account transfer
13.4.26. Track 404 pages
Behavior > Site Content > All Pages > add Page Title as secondary dimension, add filter Page Title containing Page Not Found
13.4.27. TS: Too Many Self-referrals
Reporting > Acquisition > All Traffic > Referrals, you will find the website's url is the source And the percentage is high compared to other Referral Sources. 3 reasons:
- Users sessions idle for 30 minutes and become active. This should be < 10% of traffic
- Cross domain (need to bring it down to 0 self-referral)
- A lot of traffic and sessions are created by spammers to a landing page
(most likely homepage with url parameters) with a fake referral path as the website.
You cannot use View Filter to filter out referral domain…
You can also create a segment without affecting the data. Conditions > Filter: Sessions, Exclude > Source / Medium > contains > OHG.com / referral
Ghosts :: send request to GA directly without visiting your site. Use random UA-ID and destination url.
- Identify the source domain of referral traffic Report > Acquisition > Source/Medium, change primary dimension to Source and secondary dimension to Hostname. If Hostname is not your webiste, then the Source is the ghost's referral domain. Full Referrer dimension shows referrer's domain and path.
- Create a filter under Property > View to only include Hostname that is your website
- Custom > Include > Hostname > yourmaindomain\.com|anotheruseddomain\.com
Don't do this as it will turn all referral traffic coming from that domain as direct traffic!
- Go to Admin > Account > Property > Tracking Info > Referral Exclusion List > add domain
abc.com also matches www.abc.com because domain name CONTAINS
- Usually you add your own domain to Referral Exclusion List
https://moz.com/blog/stop-ghost-spam-in-google-analytics-with-one-filter More detail instructions :: https://carloseo.com/removing-google-analytics-spam/
13.4.28. Learn
13.4.29. Checklist
- Audience > Audiences > enable Demographics and Interest Reports
- Admin > Property Settings >
- Enable Demographics and Interest Reports ga:audience:interests
- In-Page Analytics, Use enhanced link attribution
- Search Console
- Admin > Adwords Linking
- Admin > Audience Definitions > Audiences
- For a property, ga:remarketing
- Set Audience Destinations to the corresponding AdWords account
- Define your audience and give a name
- In AdWords campaign > Audience > Edit > under Targeting, Remarketing to select the list from GA
- Admin > Properties > View
- Create 3 views
- All website data, external traffic only, test view
- (no term)
- For external traffice only view, enable Bot Filtering - Exclude all hits from known bots and spiders
- (no term)
- Duplicate External Traffic Only view as Test View
13.5. Google Optimize google:optimize
- Optimize vs Optimize 360
- https://www.google.com/analytics/optimize/compare/
- Free version can only have one container in one account (container is like property in GA). Container ID is like
GTM-XXXXX
- Free version can only have one container in one account (container is like property in GA). Container ID is like
- (no term)
- A cookie is set when a user enters a variant and the user will continue to see that variant until both
_gaand_gaexpcookies are removed - (no term)
- An experiment runs on all pages unless you have targeting setup and you can select a page to run the editor to configure the variants
- (no term)
- Link to GA can compare GA metrics in variants
13.6. GA Dev Tools: Query Explorer
13.7. GA Dev Tools: Campaign URL Builder google:campaign-url-builder
- Google Analytics Campaign URL Builder
- Google Play URL Builder for advertising an Android app
- iOS Campaign Tracking URL Builder for advertising an iOS app
- Fields
- Required
- Medium, Source, Campaign ga:channel
- Optional
- Content (differentiate ads or links that point to the same URL), Term (the paid keywords)
- (no term)
- e.g.
- Medium:cpc, Source:google, Campaign:Campaign Name, Term:running+shoes
13.8. Google Search Console google:search console
- Add a new property for https if the website is upgraded from http. Stats are separated
- Google recommends creating 2 properties per root domain per schema (http/https). domainroot.com and www.domainroot.com
- Domain Property is introduced so you should create a domain property on top of all URL properties (http|https, non-www|www) for a website
- A URL Property will be automatically created if a GA account is given access. You still need to manually link Search Console from GA
- https://www.google.com/webmasters/tools/message-list
- https://support.google.com/webmasters/
- Initial HTML is crawled and indexed, and later on JavaScript is run (rendering) and a second-wave of indexing happens
- Second-wave could take several days
- js:serviceworker, local & session storage, IndexedDB, Web SQL, Cookies, Cache API
13.8.1. Performance - Search Results Report
- Total Stats
- https://support.google.com/webmasters/answer/7042828
- Total clicks
- Total Impressions
- number of impressions per click
- avg of the topmost position
- (no term)
- Queries, Pages, Countries, Devices
- impressions
- how many times your site appears on search result page
- click
- how many times people click on your website on search result page
- position
- (no term)
- More than 16 months of stats are available
- (no term)
- Refer to wp:plugin:wordpress-seo
13.8.2. Index - Coverage
- Error
Submitted URL seems to be a Soft 404
- Live Test on each URL > View Tested Page
- More Info
- HTTP Response should be 200
- Page resources
- JavaScript console messages
- Screenshot
- if content is rendered and not too many assets are blocked
13.8.3. Web Tools
- https://www.google.com/webmasters/tools/testing-tools-links
- Ad Experience Report
- google:search console:ad experience report
- (no term)
- Abusive Experiences
- (no term)
Testing Tools
- Structured Data Testing Tool
- google:search console:structured data testing tool
- Check existing structured data of a page
- Email Markup Tester
- https://developers.google.com/gmail/markup/
- Structured Data Markup Helper
- google:search console:structured data markup helper
- If you not sure how to add structured data, pick a type and page, match fields with the intended type, generate
13.8.4. Remove URL from Google Index
- This is to permanently remove a url from Google Search
- If a page is recently deleted, returning 404 and doing the following is the right way to do
- Yoast adds meta tag to 404 pages
- Google Search Console will stop showing the 404 after about a month
- But you should pay attention if the url is in sitemap but returns 404 (shown as 404 error on Search Console)
- Make the URL return 404 or 410 status
- Don't block access to that URL using robots.txt
- Return meta tag
<meta name="robots" content="noindex, nofollow"> - Return
X-Robots-TagHTTP header - Go to Google Search Console and Fetch as Google to manually tell Google update index for that URL
- Submit a removal request
- For sites you own on Google Search Console
- https://www.google.com/webmasters/tools/url-removal
- For sites that you don't own
- https://www.google.com/webmasters/tools/removals
- To check if the URL is removed
- Make a removal request and it will tell you if the URL is removed
<Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </Files>
$apacheURL = (strlen($_SERVER['REQUEST_URI'])) ? substr($_SERVER['REQUEST_URI'],1) : $_SERVER['REQUEST_URI']; $noIndexNoFollow = array( "~^subscriber-services/trucking-renewal(.*)~i", "~^lookup-subscription(.*)~i", "~^users/(.*)~i", "~^user/(.*)~i", ); if (!empty($apacheURL)) { foreach ($noIndexNoFollow as $key => $value) { $matches = array(); if (preg_match($value, $apacheURL, $matches)) { header("X-Robots-Tag: noindex, nofollow", true); } } }
13.9. Google Trends
- Compare search terms in different Google Search platforms and see their related topics and related queries
- Compare 2 keywords
hiking boots, hiking shoes- Plural vs sigular spelling
- (no term)
- Traffic change over year
A B- must contain both words
- order doesn't matter
- other words can be added before, after or in-between
- Spelling is exact
"A B"- exact phrase inside double quotation marks
- other words can only be added before or after
A + B- A or B
- center + centre + centere
- include alternative spellings. Consider each version of a word a different search
A - B- contain A but exclude B
- Special characters
- Apostrophe, single quotes and parentheses will be ignored
women's tennis world rankingyou get result forwomens tennis world ranking
- Terms vs Topics
- Term is literal match
- banana matches "banana sandwich"
- "banana sandwich" matches "banana for lunch" and "peanut butter sandwich"
- Term is literal match
- Topic is a group of terms that share the same concept in any language
- London matches "capital of the UK"
- also matches "Londres" which is London in Spanish
13.10. Google Shopping Insights
Search products (Google defines them). For US only. Nation > region > city level. By age, household income, sex, device.
13.11. Google Ads - Google AdWords, Google Keyword Planner, Google Video Advertising
13.11.1. Keyword Planner google:ads:keyword-planner
- Keyword Ideas
- Hiking, hike, backpacking, backpack
- Get keyword's search volume, avg cpc and recommendation on other relelvant keywords
- Targeting
- Geo
- Language
- Search network
13.11.2. Google Ads - AdWords
AdWords requires 1+ hour/week and AdWords Express requires 15 min/week. Express only has text ads on Search and Google Maps and doesn't require to have a website. Academy for Ads http://school4seo.com/category/google-adwords-advance-search/ https://academy.exceedlms.com/student/catalog/list?category_ids=53
13.11.2.1. Campaign Goals
Build awareness
- Targeting
- Demographics, Affinity/custom affinity audiences, Managed Placements
- Measure
- Views, Impressions, Unique users, Awareness lift, Ad recall lift
- Solutions
- six-second, non-skippable, in-stream video ads, Masthead
Influence consideration
- Targeting
- In-market audiences, Audience keywords, Similar audiences, Audience Keyword targeting, Content keyword targeting, Topic targeting
- Measure
- View-through rate, Watch time, Favorability lift, Consideration lift, Brand interest lift
- Solutions
- TrueView in-stream ads, Gmail
Drive action
- Targeting
- Display remarketing, Dynamic remarketing
- Measure
- clicks, calls, sign-ups, app installs, purchase intent lift, sales
- Solutions
- Gmail
Grow loyalty
- Targeting
- Display remarketing
13.11.2.2. Keywords Match Type adwords:keywords match types
- Default broad match
hiking tour northern california- match with queries with any order and any additional words
- match with synonyms and mispellings
- partial match is allowed "best hiking" query matches
hiking tour northern california
- Modified broad match
- + sign includes broad match but excludes synonyms
+gel batteriesmust include gel or its synonym, batteries may not presentgel +batteriesmust include batteries or its different forms, gen may not present
- Phrase match with double quotes
- any queries match
"hiking tour northern california"in any order with additional words - Exact match and only these words
[hiking tours northern california]- Negative match
-dog,-wallpaper,-screensavers,-picturesto ignore queries that have this negative word- (no term)
- google:search operators are ignored
- (no term)
- https://support.google.com/adwords/answer/7476658
- (no term)
- https://support.google.com/adwords/answer/2472708?hl=en
13.11.2.3. Negative keywords
Negative broad match :: ad won't show if the search contains all negative keyword terms, even if the order is different. running shoes
Phrase match :: ad won't show if the search contains the exact keyword terms in the same order.
Exact match :: ad won't show if the search contains the exact keyword terms, in the same order, without extra words.
https://komarketing.com/blog/200-plus-negative-keywords-to-consider-for-b2b-ppc/ hawaii career careers "new balance sale" "sale in new brunswick"
13.11.2.4. Campaigns, Ad groups
- Campaign
- It's usually for a type of product you sell. If budget or location targeting is changed, create another campaign
- Setting
- frequency capping, start/end dates, budget, goal, locations, language, bidding strategy, devices, content exclusions, IP exclusions, dynamic ads feeds, ad rotation, campaign URL options
- Frequency capping
- level can be set ad, ad group and this campaign
- Drafts & Experiments
- A draft inherits all settings from a campaign and almost any ads and ad groups can be changed. After a draft is made, it can be applied to the original campaign or creat an experiment to test how changes perform against your original campaign
- Limitation
- some features and reports that are not available for drafts, and some features aren't supported by experiments
- Experiments
- while multiple drafts for a given campaign, only one draft can run as an experiment at a time
- Specify how long to run and how much original campaign's traffic (and budget) to use for the experiment
- Later, an experiment can be applied to the original campaign or convert the experiment to a new campaign
- Ad group
- It holds one or more ads which target a shared set of keywords (theme) or a set cost-per-click (CPC) bid
- To show ads that are relevant to the searches of people you're trying to reach, bundle related ads together with related keywords into an ad group. That way, all of your related ads can be shown to customers searching for similar things
- For each ad group, use keywords related to that theme. Consider also having your ads mention at least one of your keywords in its headline
- bidding, ad rotation, ad group targeting, landing pages, keywords, placements, demographics, audiences, narrow targeting, observations, automated targeting
- Ad Group Targeting
- audiences, demographics, keywords, placements, additional observations, automated targeting
- Ad
- URL, creative, Search Ad (title, URL, description)
Organize account with ad groups
Linking to someone's AdWords account lets that person advertise your locations with location extensions. Linking to someone's Merchant Center account lets that person advertise your products with local inventory ads.
Dynamic Search Ads
- A campaign can enable DSA and the default ad group (1st ad group when a new Search campaign is created) is a dynamic ad group
- Standard Ad Group can be added as well.
- Only Dynamic Search Ads can be added to this dynamic ad group. For each ad, provide just a description field
- Dynamic Ads Targets are created to categorize webpages of your destination website (domain). And you can further select categories to target the Dynamic Search Ads created in this dynamic ad group
13.11.2.5. Placement
Locations on the Display Network where your ad can appear. Examples include relevant websites and apps that partner with Google to show ads.
13.11.2.6. Quality Score, Ad Rank, Ad Position
- Quality Score
- Relevancy
- someone searches "snowboard rentals" and your ad uses keyphrase "Snowboard rentals Tahoe"
- Click-through rate (CTR)
- clicks over views
- Account history
- AdWords account
- Landing page relevancy
- content match what people search (bounce rate)
- adwords:ad rank
- Ad Rank = Quality Score x Maximum bid x ad formats.
13.11.2.7. Bidding Strategy - Media Cost Model
Campaign goals decide on which media cost model you should use.
Use manual bidding first and after the campaign is already getting at least 15 conversions and 15 clicks per month, automated bidding can help:
- Maximize clicks
- Good for advertisers just starting off with online marketing who are most interested in getting customers to their website.
- Ad revenue
- Target return-on-ad-spend (tROAS). Good for advertisers who know the exact value of each conversion.
- Ad conversions
- Good for advertisers looking to drive conversions. "auction-time bidding"
- Enhanced cost per click (ECPC)
- described later
- Maximize conversions
- max number of conversions. May use up all daily budget for one conversion
- Target cost per acquisition (tCPA)
- max number of conversions while maintaining avg CPA. Good for multiple campaigns with various CPA. Some bids might cost higher than the corresponding target you set but some are lower in order to make up the avg cost is within your set target. CPA is the average amount you pay for a conversion.
Cost per Click (CPC) is determined by max bid + Quality Score + Ad Rank and compare those against your competitors. Drive traffic.
Set Max. CPC to = your profit x commission for Google x your conversion rate
$100 x 0.3 x 1% = $.30 1% conversion rate is based on 1000 views of the page and 10 people buy
- Manual CPC
- each keyword or Ad Group would have the same bid
- Automatic CPC
- there're some bid strategies under this category. Max CPC cannot be set
- Enhanced CPC (ECPC)
- relies on Google’s own historical data to help you predict where and when to adjust bids you set manually in order to get more conversions. Its CPC is constrained by Max CPC you set
- e.g. if a campaign’s performance looks promising it will automatically raise bids to ‘capture’ more results (for less money). Similarly, it will also drop bids if necessary to help you save on wasted ad spend if performance starts to slide.
- it's better to turn it off initially and let AdWords collect data and further turn it on
- individual CPC might exceed Max CPC you set, but avg CPC is below Max CPC
CPC, CPA and CPM are 3 bidding strategies but each one can be further optimized based on bid modifiers which are rules that raise or lower bids based on:
- Geographic locations
- although more traffic from mobile but conversion rate is higher on desktop
- Dayparting
3 phases
- Brand awareness
- Let the world know about you. Requires larger budget due to the longer path to conversion and the scale at which you try to reach people.
- Influence consideration
- Encourage customers to explore you.
- Driving action or sale
- lower funnel usually requires less budget.
- CPA - boost sales
Cost Per Action (or conversion)
- CPM, vCPM - brand awareness
- Applies only to the Display Network (along with remarketing campaigns, too). Here you pay a cost (like a few cents or dollars) per one thousand impressions
- viewable cost per thousand impressions. Cost a bit more than CPM.
- CPV - boost video views
cost per view for video or click on it.
- Bid Simulators
https://support.google.com/adwords/answer/2470105
Click Campaigns, Ad groups, or Keywords. If you're in Campaigns, the icon is in the "Daily budgets" column. If you're in Ad groups or Keywords, the icon is in the "Default max. CPC" column.
Some reasons why bid simulators do not show
- Search and Display campaigns that use Manual CPC, Enhanced cost-per-click, and Target CPA bid strategies.
- campaign has reached or nearly reached its daily budget at least once in the last 7 days
- a campaign, keyword, ad group, or product group was recently added, or didn't receive many impressions or clicks in the last 7 days.
- campaign uses shared budgets
- First-page bid estimate
For CPC, AdWords can show estimates for showing your ad on:
- First page
- at the top of first page
- First position
AdWords > Keywords > Add columns > Attributes > Est. first page bid, Est. top of page bid, Est. first pos. bid
- Bid Adjustments
- Bid adjustment is available in adwords:targeting
- Clicking “Advanced bid adj.” opens the Interactions page, where you can set your bid adjustments for calls.
- Decrease a bid by 100% is to completely exclude the ad from a targeting setting
- Daily budget
Divide by 30.4 that is avg number of days per month.
Delivery method
- Standard
- Accelerated
13.11.2.8. Networks, Campaign Types, Ad Formats
Search network is likely to bring more traffic to your site. Display network is for driving awareness e.g. promote for a new website or something that has a new concept
- Search Network
Google Search, Shopping, Maps and Google Play for websites and Google apps. Also Google Search Parnters. Search Parnters :: non-Google websites, as well as YouTube and other Google sites. CTR for ads on search partner sites doesn't impact Quality Score on google.com. It is included by default. To remove it AdWords account > Settings > select campaign > Networks > uncheck Include Google search partners
Ad formats
- text ads
- Search Network and Search Partners
- ad extension
- adwords:ad extension
- (no term)
- dynamic search ads
- shopping ads
- Google Search, Shopping, Search Partners
- image ads
- Display Network, Search Partners but not Google Search Network
- video ads
- Display Network, Search Parnter Networks but not Google Search Network
- app promotion ads
- Search Network, Display Network
- call-only ads
- Search Network
- Display Network
Shows ads on other websites, other apps, YouTube, Gmail, Blogger. New or changes can take 12-24 hours to apply.
Display Network can target audiences by interests, topics, placements. adwords:targeting
Ad formats
- Responsive ads
- responsive can fit in all available sizes and can transform into text or image ads.
- Landscape: ideally sized at 1200 x 628
- Square: ideally sized at 1200 x 1200
- A short headline (25 characters or fewer) which appears in tight ad spaces. It may or may not appear with a description.
- A long headline (90 characters or fewer) which appears instead of the short headline in larger ads. In some cases, it may be shortened with ellipses. It also may or may not appear with a description.
- A description which may appear with the headline and invite users to take action. It also may be shortened with ellipses.
- Image ads
- Creat as many sizes and formats as possible in order to reach on Display Network.
- (no term)
- rich media ads
- Engagement ads
- image and video ads on YouTube and across the Display Network
- Gmail ads
- expandable ads on the top tabs of people's inboxes. Can't use affinity, remarketing or remarking-based audiences. May interests and domain targeting (receive emails from).
- Video ads
- remarketing on people who view your YouTube videos or channel as they browse Display Network.
- (no term)
- app promotion ads
- (no term)
- adwords:ad extension
- Shopping
- Video
- TrueView in-stream ads
- YouTube and across Display Network sites, games, or apps.
- TrueView video discovery ads
- YouTube.
- Bumper ads
- 6 seconds or less.
- Universal App
Downloads of your app
13.11.2.9. Targeting options adwords:targeting
- Keywords
- try to include between 5 and 20 keywords per ad group
- Audience
- Show ads to people likely to be interested in these keywords and also on webpages, apps, and videos related to these keywords
- Content
- Only show ads on webpages, apps, and videos related to these keywords e.g. webpage's concepts.
- (no term)
- Audience
- Interests and habits (Affinity and custom affinity)
types of websites not webpages the audience have visited. This is for running TV ad
- Affinity is more general.
- Custom Affinity can have these
- Interests entered as keywords
- URL
- places
- apps
- Custom Affinity is only for Display Campaign
- Life evnets
- engage with viewers on YouTube and Gmail
- Researching or planning (In-market and custom intent)
- likely buyers
- (no term)
- Remarketing
- Similar Audience
- better to create another campaign. Not all custom audience list can have a similar audience
- (no term)
- Audience can be added on a campaign or an ad group
- Demographics
- age, gender, income
- Topics
- topics a webpage has e.g. central theme
- Placements
- specific websites your ad will show on. May also be specific YouTube videos, channel.
- (no term)
- Ad schedule
- Locations
- majority of consumers want ads customized to their city, zip code, or immediate surroundings
- (no term)
- Settings
- Devices
- Languages
- IP exclusions
- Content exclusions
- use Data feed to show different ads based on audience who have visited certain webpages on your website with custom parameters tagged. Refer to ga:dynamic remarketing for tagging.
13.11.2.10. Ad Group Automated Targeting
- Only for Display Network, under Ad Group > Settings > Automated targeting
- default
- broader match. e.g. if your keyword is “pens,” conservative targeting may extend to “felt-tip pens" and “ballpoint pens,” but aggressive targeting might show your ads in contexts related to “whiteboard markers” or “mechanical pencils”—if there’s data to suggest that those keywords will lead to conversions. Plus, automatic targeting works for remarketing. Aggressive targeting is available for all Display Network campaigns with at least 15 conversions per month. It costs a little bit more than Conservative.
- When to use automatic targeting
- Find more customers
- Identify the best targeting to reach your most likely customers
- Increase reach without increasing bids or cost per customer
13.11.2.11. Ad Formats in networks
- text ad
adwords:text ad Goal :: specific, actionable, and relevant. Best practice. Required *
- *Headline
- consider including at leat one keyword. 2 fields up to 30 characters each. Fields are combined with hyphen.
- *URL
- Path field appear as www.example.com/Indoor-Plants but actual is different.
- *Description
- 80 characters
Some good phrases are:
- Shop now, Buy now, Get an instant quote online, See pricing
13.11.2.12. Ad extension
- Basics adwords:ad extension
Extension appears with ads on the Search Network. Some extensions might also appear with ads on the Display Network.
Extensions are free to use. List of extensions.
Factors determining when they show:
- Your Ad Rank, which combines your bid, the quality of your ad and landing page, Ad Rank thresholds, context of the person’s search, and expected impact of extensions and other ad formats. AdWords requires a minimum Ad Rank (factoring in your extensions) before showing extensions. So, you may need to increase your bid or your ad quality (or both) in order for your extensions to show.
- The position of your ad on the Google search results page. There’s a limited amount of space available above Google search results for ad extensions, and ads in higher positions get the first opportunity to show extensions. Ads in lower positions generally won’t have more extensions than ads in higher positions. Plus, the AdWords system generally won’t allow ads in lower positions to get more incremental clicks from extensions than the incremental clicks they’d get from moving up to a higher position. To show ads in higher positions, generally you need to increase your ad quality, bid, or both.
- Other extensions you’ve enabled. In each auction, we'll generally show your highest performing and most useful combination of eligible extensions and formats. You will not be able to get a combination of extensions which gives more expected click-through-rate (CTR) than the expected CTR of a higher ad position.
Both ad formats and ad extension could help your ad's Ad Rank!
Extensions can be set on campaign level.
- Location extension
Works on both Search and Display Network including Google Maps. https://support.google.com/adwords/answer/7040605
Encourage people to visit your business by showing your location, a call button, and a link to your business details page—which can include your hours, photos of your business, and directions to get there.
- Call extension
Has some limitation on Display Network.
- Message
Encourage people to send you text messages from your ad.
- Price
Showcase your services or product categories with their prices, so that people can browse your products right from your ad.
- Promotion
Highlight specific sales and promotions across your ads (e.g., 30% off rose bouquets)
- App
Available globally for Android and iOS mobile devices, including tablets.
- Sitelink
Show six to eight additional links to different pages on her site. e.g. store hours, location details, and popular bouquets
- callout
Callouts can improve your text ads by promoting unique offers to shoppers, like free shipping or 24-hour customer service. When customers see your ads, they get detailed information about your business, products, and services.
2 to 6 callouts show in addition to the text of your ad. Ads with callout extensions can show at the top and bottom of Google search results. When callout extensions show, they appear below your ad copy.
Free Shipping · 24-7 Customer Service · Price Matching
Callout text is limited to 25 characters in most languages, or 12 characters in double-width languages (like Chinese, Japanese, and Korean).
- Structured snippet
Highlight specific aspects of your products and services with structured snippets extensions. Structured snippets show beneath your text ad in the form of a header (ex: "Destinations") and list of values (ex: "Hawaii, Costa Rica, South Africa").
Services: Tech Support, E-Waste Recycling, Computer Repair
List of headers Amenities Brands Courses Degree programs Destinations Featured hotels Insurance coverage Models Neighborhoods Service catalog Shows Styles Types
It can show up to 2 headers at a time, while ads that show on mobile and tablet devices will only show one header. AdWords algorithmically decides the best header or combination of headers to show, so it’s best to add as many headers as possible that are relevant to your business.
- Universal extensions
Sitelink, Callout and Structured Snippets are called universal extensions that fit all business objectives.
13.11.2.13. Ad customizers
13.11.2.14. ValueTrack URL parameters
These parameters can be used in Final URL and Tracking Template. Here's an example of Tracking Template
{lpurl}?matchtype={matchtype}&device={device}
All parameters https://support.google.com/adwords/answer/6305348
{keyword}- For the Search Network: the keyword from your account that matches the search query, unless you are using a Dynamic Search ad, which returns a blank value. For the Display Network: the keyword from your account that matches the content.
Parallel tracking should be enabled :: AdWords account > All campaigns > Settings > Account Settings > Tracking > Parallel tracking
13.11.2.15. Supported Ad Sizes
Square and rectangle 200 × 200 Small square 240 × 400 Vertical rectangle 250 × 250 Square 250 × 360 Triple widescreen 300 × 250 Inline rectangle 336 × 280 Large rectangle 580 × 400 Netboard
Leaderboard 468 × 60 Banner 728 × 90 Leaderboard 930 × 180 Top banner 970 × 90 Large leaderboard 970 × 250 Billboard 980 × 120 Panorama
Skyscraper 120 × 600 Skyscraper 160 × 600 Wide skyscraper 300 × 600 Half-page 300 × 1050 Portrait
Mobile 300 × 50 Mobile banner 320 × 50 Mobile banner 320 × 100 Large mobile banner
Most popular for responsive ad (covers 95% in Display Network) :: 300x250, 728x90, 1600x600, 320x50, 300x600
13.11.2.16. Conversion tracking
- Conversion tracking with Google Analytics
- https://support.google.com/google-ads/answer/2375435
- google:ads:link GA
- google:ads:auto-tagging
- If GA has linked to the AdWords account and goals are defined on GA, AdWords conversion tag may not be necessary to put on the website
- Tools > Measurement > Conversions > New > Import Google Analytics
- New stats are imported when Import is clicked. Conversion data can be seen on AdWords within 2 days or 9 hours behind GA shows goal conversion stats. Historical data before the import won't be included
- If GA goal newly created or a GA goal is newly imported to Google Ads, it may take about 30 minutes for it to appear on Google Ads
- Conversion tracking using Google Ads conversion tag (on websites, mobile apps, call tracking)
- https://support.google.com/adwords/answer/1722054
Advantage about Google Ads conversion tracking
- View-through conversion
- Measures how many visitors saw ads on Google Display Network and YouTube Video but did not interact/click but they later complete a conversion on your site
- Google Ads conversion tracking is required. Cannot simply setup goals on Google Analytics
- https://support.google.com/adwords/answer/1722021
- (no term)
- Conversion tracking data (columns) seen on Google Ads once conversion tracking is set up
Use global site tag on website same as remarketing
<script async src="https://www.googletagmanager.com/gtag/js?id=AW-CONVERSION_ID"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'AW-CONVERSION_ID'); // or you just add this to your existing GA global site tag // If you want to manually send an event to mark a conversion: // gtag('event', 'conversion', {'send_to': 'AW-CONVERSION_ID/conversion-unique-specific-hash-id'}); </script>
- Phone adwords:conversion:phone
Call Extension or call-only ad, enable Call Reporting and you have to change "count conversions as" to a new Conversion tracking instead of the default "Calls from ads"
https://support.google.com/adwords/answer/6095882
Measurement > Conversions > Add new Phone calls - Calls from ads using call extensions or call-only ads
After that, go back to your existing call extension or call-only ad or when you create them from now on Select the conversion action you just created as Count conversion as
- Calls to a phone number on your website
- https://support.google.com/adwords/answer/6095883
- Measurement > Conversions > Add new Phone calls - Calls to a phone number on your website
- It prompts the phone snippet based on your option. Sample in header
<script async src="https://www.googletagmanager.com/gtag/js?id=GA_TRACKING_ID"></script> <script> window.dataLayer = window.dataLayer || [ ] ; function gtag(){dataLayer.push(arguments);} gtag( 'js', new Date () ) ; gtag( 'config', 'GA_TRACKING_ID'); gtag( 'config', 'AW-CONVERSION_ID'); </script> <script> gtag('config', 'AW-CONVERSION_ID/CONVERSION_LABEL', { 'phone_conversion_number': '1-650-555-5555' }); </script>
If you select "Don't enter a number"
gtag('config', 'AW-CONVERSION_ID/CONVERSION_LABEL', { 'phone_conversion_number': '1-650-555-5555', 'phone_conversion_callback':function(formatted_number, mobile_number) { // formatted_number: number to display, in the same format as the // number passed to 'phone_conversion_number'. // (in this case, '1-650-555-5555') // mobile_number: number formatted for use in a clickable link // with tel:-URI (in this case, '+16505555555') var e = document.getElementById("number"); e.innerHTML = ""; // e.href = "tel:" + mobile_number; e.appendChild(document.createTextNode(formatted_number)); };, 'phone_conversion_options':timeout=20;cache=false' });
Adding this will get a telephone number and replace the contents of all spans of the given class
phone_conversion_css_class: 'number'
<span class="number">1-800-123-4567</span>
- Calls to a phone number on your website
- Attribution Modeling
It is only available for clicks on Search Network and Shopping ads on Google.com. Which ad click makes the final conversion?
Data-driven :: only available for search campaigns with 15k+ Google.com search clicks and 600 conversions per conversion type over 30 days. Position-based :: 40% of credit to both first and last clicked ads and corresponding keywords, remaining 20% spread out across the other clicks Time decay :: More credit to clicks that happened closer in time to the conversion Linear :: Credit for the conversion is shared equally across all clicks on the path
13.11.2.17. Remarketing Audience, Customer Match
- One Google Ads account has one AdWords tag as in Audience sources
- May get audience from the past 30 days or start from 0 audience
- Other sources include Google Analytics, YouTube
- https://support.google.com/adwords/answer/2453998?hl=en
- Rules and Templates
- Rules
- https://support.google.com/adwords/answer/6297497
- Standard parameters include URL, Referrer (e.g. www.google.com coming from search or anothersite.com), parameters sent through the event snippets and attributes from Google Merchant Center
- Path and URL query parameters can be targetted. But not
#
- Template
- https://support.google.com/adwords/answer/6297549
- Visitors of a page
- Add multiple conditions and if any condition met, then add to the list
- Visitors of a page who also visited another page
- To create a list of people who visited pages A, B, and C, first create a list of people who visited pages A and B using this template. Then, using the "Visitors of a page" template, create a list of people who visited page C. Then, create a custom combination using these two lists.
- Only on Display Network
- Visitors of page who did not visit another page
- When you create a list using this template, you narrow your audience. To be on your list, visitors need to visit the page defined in the first rule AND not visit any of the pages defined in the second rule. This is what makes this template different from the "Visitors of a page" template. When you create a list using the "Visitors of a page" template, you can add different conditions to the rule, but people who match ANY of the conditions (as opposed to ALL of the conditions) will be added to the remarketing list.
- Only on Display Network
- Visitors of a page during specific dates
- When creating lists with specific dates, lists' membership duration may also determine when people are removed from the list. Visitors are added to the list if they've visited the pages on the selected dates, and will stay on the list for the amount of time specified by the membership duration.
- Visitors of a page with a specific tag
- Awhile ago, an airline added a remarketing tag to sections of their website about popular routes. Creating a rule based on the URL for those sections would be too complex using rules. The airline could create a remarketing list of people who visited sections of the website about the popular route by using the "Visitors of a page with a specific tag" template, and selecting the tag that was implemented a while ago on those pages.
- When you use the "Visitors of a page with a specific tag" template, you can select existing AdWords conversion tracking tags as well. Selecting a conversion tracking tag can be helpful if you haven't been able to add the remarketing tag to the conversion page but still want to reach or exclude this audience.
- Visitors of a page
- Custom Combination List
- Customer Match
It's for Search, YouTube and Gmail Networks.
- Upload a list as an Audience List to match Google accounts.
- Email, Phone, First Name, Last Name, Country, Zip, and Mobile Device ID (multiple email and postal columns are allowed)
- https://support.google.com/adwords/answer/7474166
- Similar audience targeting based on your Customer Match audiences is available for YouTube and Gmail.
- Upload a list as an Audience List to match Google accounts.
13.11.2.18. Reports
- System Reports
Search terms :: When you use broad-match keywords (the default setting), your ads can appear when someone searches for a variation of your keyword, like a similar phrase or related word. To see a list of searches that have triggered your ad, use the Search terms report. You can use this report to identify relevant terms that are driving traffic to your website, and then add them as new keywords. Or, if any of the keywords are irrelevant to your business, you can add them as negative keywords so they won't trigger your ads.
Paid and organic report :: how people got to you — comparing Google’s free organic search results to your paid AdWords ads. Learn the ways customers are looking for products and services like yours and update your own keyword list or create new ad groups to directly target them. In order to access this report you’ll need to link Google Webmaster Tools to Google AdWords. AdWords > Linked Accounts.
User locations :: physical locations Geographic :: Location of interest and physical locations.
Landing page experience :: located under Keywords > a keyword > column Status to review Ad Rank
Wrench icon > Measurement > Search Attribution :: Once you've set up conversion tracking, you'll have access to a handy set of reports about your conversions. Attribution reports show you the paths customers take to complete a conversion, and attribute the conversion to different ads, clicks, and factors along the way.
Return on investment ROI is calculated with this formula: (Revenue - Cost) / Cost. Example: If your ad resulted in $1,200 of sales for a product that cost $600 to make, and your advertising cost was $200, then your ROI is [$1,200 - ($600 + $200)] / ($600 + $200) = 50% ROI.
Segments For any reports, you can click on 3 bars next to filter to create segment
- Network
- Device
- Time
- Click type
- Conversions
- Top vs. Other
- Keyword text
Auction Insights :: how well you did in auctions compared to other advertisers that compete with you? Available to Search Network and Shopping. Keywords > Auction Insights
Reach Metrics (only in Campaigns):
- Unique users
- same user but across multiple devices
- (no term)
- Unique cookies
- Unique viewers
- same as cookie but for Video ads only
- Frequency
- an estimate of the average number of times a unique cookie was exposed to your ad over a given time period.
Low CTR but a lot of impressions :: If the goal is max sales, try to lower the Max CPC.
High conversion rate and low CPC :: increase Max CPC for this keyword.
- Custom Reports
Custom reports are made on Account level. Filter to a specifc campaign if necessary.
e.g. Row:Campaign Columns:All stats include Phone impressions, Phone calls and phone-through rate (Call details).
Individual row and column can be further filtered.
If you want to break down to individual goals, use GA.
GA does not show AdWords phone conversion.
- Breakdown Conversions inside Google Ads
Breakdown conversions #1 (User Location & Ad Groups)
- Rows
- Ad group, Most specific location target, Region, Conversion action, Device, Day
- Columns
- Conversions
Breakdown conversions #2 (Search Term & Ads)
- Rows
- Ad, Search term, Conversion action, Device, Day
- Columns
- Conversions
Breakdown conversions #3 (Hours of Day & Ad Groups)
- Rows
- Ad Group, Hour of Day, Conversion Action, Device, Day
- Columns
- Conversions
Breakdown conversions #4 (Extensions & Ad Groups)
- Rows
- Extension, Ad Group, Day, Conversion Action, Device, Day
- Columns
- Conversions
- Breakdown Conversions inside Google Ads
13.11.2.19. Autotagging google:ads:auto-tagging
- Always enable autotagging in AdWords
- Previous view > Gear icon > Account Settings > Preferences > Tracking > Auto-tagging
- Auto-tagging is turned off by default. Settings on the left page menu > Account Settings along the top > Auto-tagging. On if "Tag the URL that people click through from my ad" is checked
- https://support.google.com/analytics/answer/1733663
If the final URL contains any of the following, manual tagging is used:
- utm_source, utm_medium, utm_campaign, utm_content, utm_term
When auto-tagging is used, gclid=abc parameter is appended to the url. Auto-tagging only works with Google Analytics e.g. other 3rd party tools which only use UTM parameters.
You can add UTM parameters and with auto-tagging on, UTM parameters and gclid will be in the URL. You can also let GA take the UTM parameters you specifiy instead of the values that gclid can bring.
- https://support.google.com/analytics/answer/1033981?hl=en
- Don't include hash in the final URL with auto-tagging on! as gclid URL parameter will not be caught as url parameter.
13.11.2.20. Automated Rules
- They are added to a campaign (e.g. enable and disable a campaign), an ad group, an ad or extension and others based on time or performance
- Manage Rules under Reports in the top bar between Go To and Tools.
- https://support.google.com/google-ads/answer/2497710?co=ADWORDS.IsAWNCustomer%3Dtrue&hl=en
13.11.2.21. Third party impression tracking
Submit a form to tell Google you want a 3rd party impression/view/skip/DCM Enhanced YouTube Pixel tracking.
Only available to Display and Video campaigns.
13.11.2.22. Accounts google:ads:account
- Either users in Normal Account or users in Manager Account
- will have access to all campaigns/settings of the Normal Account
- To link a Normal Account to a Manager Account
- so that the Manager Account can manage the Normal Account. On the Manager Account, on the left Accounts > Management > + and Link existing accounts, copy and paste the Normal Account's Customer ID, Send Request. Go to the Normal Account > Tools & Settings > Account access > Managers > approve. You can set the Manager Account as Administrative owner while on Normal Account, that way the Manager Account can create/approve/revoke access for any others
- Normal account
- also called managed account or individual account
- 1 Customer ID
- 123-123-1234. Click on icon
?in the top right corner - Multiplue Users
- who may modify the whole parent account
- Multiple Manager Accounts (Managers)
- who can modify the whole parent account
- To link a Manager Account to this account
- go to the Manager Account and add this account's Customer ID
- Campaigns
- 1 Payments Account ID
- 1 Payments Profile:: https://pay.google.com/
- Each Google Account has a Payment Profile. A Normal Account can have its own Payments Profile
- For Google Account, add users to Payments Profile with permissions when Payments Profile Account Type is Business
- For Normal Account, add contacts to receive email notifications only
- https://support.google.com/google-ads/answer/7058401
- You can set it up on the Manager Account and link it to this account, then go to Billing of this account to add the Manager Account's Payments Profile
- The 1st step requires to contact Google's online specialist
- Manager account
- One Google Account (email) can only see 20 Normal Accounts. You need a Manager Account with users to access more Normal Accounts
- Create one
- use a different email that is not used for any Normal Account or Manager Account. https://ads.google.com/home/tools/manager-accounts/
- 1 Customer ID
- 123-123-1234. Click on icon
?in the top right corner - Multiple Users
- Multiple Manager Accounts (Managers)
- Views
- Overview
- Recommendations
- Accounts
- Campaigns
- Change History
- Partners program
13.11.2.23. Link a Google Analytics Property and Google Ads accounts google:ads:link GA
- Linking a GA property to Google Ads account can help you analyze customer activity on your website after an ad click or impression
- The same Google Account should have Edit permission for the Analytics property and Admin access for the AdWords manager account
- Sign in to Google Ads account
- Click the tools icon in the upper right corner, under Setup, click Linked accounts
- Under Google Analytics, click Details
- Search the GA property you want to link e.g.
UA-xxxand click Link. If the property is not in the list, check if you have GA Edit permissions After linking, you can how many views of that GA proerpty are linked. Click on the Views to select which views you want to link. 2 settings for each view
- Link
- make Google Ads click and cost data available in GA and GA goals and transactions available from import into Google Ads
- Import site metrics
- import site engagement metrics from GA. It's used to show site engagement metrics in GA reporting columns on Google Ads
- Need to add GA data (reporting columns) to Google Ads reports
- https://support.google.com/google-ads/answer/2617364
13.11.3. Make ads
13.11.4. Google Video Advertising
https://support.google.com/displayspecs AdWords :: https://support.google.com/adwords/answer/2375464
Google Preferred :: Goolgle serve your video ads on YouTube videos that are the top 5% content across highly popular channels, playlists or video collections. Videos are divided into several categories.
Desktop and Mobile Video Mastheads :: https://support.google.com/displayspecs/answer/6244544. YouTube homepage 100% share of voice, target country and device and device.
DoubleClick Bid Manager :: programmatic buying platform
13.11.4.1. TrueView in-stream ads
- TrueView in-stream
In-stream ads play before or during another video from a YouTube partner. Viewers see five seconds of your video and then have the choice to keep watching or skip it.
You pay when a viewer watches for at least 30 seconds or to the end of the video (whichever is shorter) or clicks on a card or other elements of your in-stream creative.
Interactive Features you can add:
- Call-to-action overlays
- End screens
- Cards
- Auto end screens
- TrueView video discovery
Video discovery ads appear alongside other YouTube videos, in YouTube search pages, or on websites on the Google Display Network that match your target audience.
You pay only when a viewer chooses to watch your video by clicking on the ad.
13.11.4.2. Measure & Solutions
Awareness
- Views, Impressions, Unique users, Awareness lift, Ad recall lift
- six-second, non-skippable, in-stream video ads
- Masthead
Consideration
- View-through rate, Watch time, Favorability lift, Consideration lift, Brand interest lift
- TrueView in-stream ads
Action
- Clicks, Calls, Sign-ups, App installs, Purchase intent lift, Sales
13.11.4.3. Video Remarketing
Remarketing lists :: YouTube related actions or Google Search Display behaviors. Views, likes, commenting, sharing and subscribing, as well as visiting channels, videos or even mastheads
These lists can be used in existing, new and even Search and Display campaigns.
In AdWords :: Linked Accounts > Youtube channel > you own or someone else owns
YouTube users :: viewed any video from a channel, viewed certain videos, viewed any video (as an ad) from a channel, viewed certain videos as ads, subscribed to a channel, visited a channel page, liked any video from a channel, added any video from a channel to a playlist, commented on any video from a channel, shared any video from a channel
Features on AdWords after linking YouTube channel to an AdWords account
- View counts and calls-to-action
- view ad completion rates of videos, create CTA overlays on videos from linked channels.
- Remarketing
- create lists based on viewers' past interactions on linked channels
- Engagement
- view earned actions metrics from videos and video ads from linked channels.
Multiple AdWords accounts can be linked to a YouTube channel.
Metrics
- View rate
- % of people who choose to watch video ads after seeing the video ad's thumbnail.
- Avg. CPV
- cost per view
- (no term)
- CTR
- (no term)
- Cost
- Earned actions
- happen when a viewer watches a video ad and then takes a related action on YouTube.
- Earned views
- watch subsequent videos on the channel or Watch pages
- Earned subscribers
- happen when a viewer subscribes to your channel.
- YouTube Analytics
- Audience retention report (e.g. relative audience retention for a video compared to the YouTube average for similar videos)
- calls-to-action and other interactive elements, # of shares, where/when you gain or loose subscribers
Contact Google representative to conduct a Brand Lift survey to show metrics in business meaning :: lifts in awareness, ad recall, brand interest, consideration, brand favorability, purchase intent. You will need to provide several of your competitors in order to conduct the survey.
13.11.4.4. YouTube Advertising
https://www.youtube.com/yt/advertise/ Learn how to make a video https://creatoracademy.youtube.com/page/welcome
13.11.5. Google Rich Media Gallery
Templates, Learn, Formats/Ad Format Gallery
13.12. Google Tag Manager google:gtm
- https://support.google.com/tagmanager/
- https://tagmanager.google.com
- Lynda
- Refer to chrome:wasp
- a group of containers. Can be linked to google:marketing platform
- Container
- a website
- Inside a container
- Tags, Triggers, Variables, Folders
- Edit, Approve (approve the edits), Publish (highest permission)
13.12.1. Code
Container > Admin > Install Google Ad Manager
<head> <!-- as high in the <head> as possible --> <!-- If JS libraries are used for GTM, then put this after the libraries are loaded --> <!-- Define dataLayer for GTM to define Variables for triggers or tags dataLayer will persist as long as the visitor remains on the current page --> <script> dataLayer = [{ 'pageCategory': 'signup', 'visitorType': 'high-value' }]; <!-- a diffrent custom parameter can also be used in GTM --> myNewName = []; </script> <!-- Google Tag Manager --> <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-CONTAINER-ID');</script> <!-- End Google Tag Manager --> <!-- Change to use custom parameter myNewName instead of default dataLayer --> <!-- <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','myNewName','GTM-CONTAINER-ID');</script> --> <!-- dataLayer is defined after GTM is loaded --> </head> <body> <!-- immediately after the opening <body> --> <!-- Google Tag Manager (noscript) --> <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-CONTAINER-ID" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript> <!-- End Google Tag Manager (noscript) --> <!-- After GTM is loaded, should use push() API to modify dataLayer --> <a href="#" name="button1" onclick="dataLayer.push({'event': 'button1-click'});" >Button 1</a> </body>
After that, click SUBMIT to publish the change
13.12.2. Tag
- Tag Sequence
- Choose another tag to run before or after the current tag
- Hence, a tag might not need a trigger
- Types
Custom HTML you can paste custom JavaScript
<script> console.log('local hour: {{Local Browser Time-hour}}'); </script>
- Enable
Support document.writeif it doesn't have external sources
- Enable
13.12.3. Variables
- Variables can be referred to in Tag and Trigger as value using double curly brackets e.g.
{{Page URL}} - Built-in variables without any tags. These are not all built-in variables:
- All built-in variables
- https://support.google.com/tagmanager/topic/7182737
- (no term)
- Event
- (no term)
- Page Hostname
- (no term)
- Page Path
- (no term)
- Page URL
- Click URL
- after a link is clicked
- Variables come with a certain tag type
TransactionTotal = {insert cart checkout total here}- GA Custom dimensions
- Google Analytics Settings
- Used to config multiple GA tags
- Some events do not use the Settings' content group for the events
- https://support.google.com/tagmanager/topic/9125128
- Constant Variable
dataLayer.push({"Data Layer Name": "value"})google:gtm:user-defined variable:data layer- Version 1 ::
dataLayer.push( "a.b.c": "value") - interpreted as
{ "a.b.c": "value"} - Version 2 ::
dataLayer.push({"a.b.c": "value"}) - interpreted as
{a: {b: {c: "value"}}}
- Version 1 ::
- Put e.g.
document.titleas Global Variable Name and this GTM JavaScript Variable is the same as that variable these vars can be available as soon as Page View. Refer to trigger
function() { return new Date().getHours(); }
13.12.4. Trigger
- Trigger controls under which condition a Tag is used/inserted. A Tag always requires at least one trigger
- Types
- Page View
- 2 types
- Page View
- fire immediately when the browser begins to load a page
- DOM Ready
- Pageview-based tags that interact with the DOM to populate variables should use this
- Window Loaded
- fire when page has fully loaded
- (no term)
- Click
- Just Links
- The following example fire an event to track outbound clicked link
- Create a Tag to be fired after GTM event is triggered
- (no term)
- Universal Analytics
- Type
- Event
- Category
- link clicks
- Action
- outbound
- Label
{{Click URL}}- Tracking ID
- UA-XXX
- Create a trigger for the Tag
- Trigger Type
- Click - Just Links
- (no term)
- Click URL does not contain yourwebsitedomain.com
- (no term)
- This trigger fires on
All Link Clicks
- Custom Event
- google:gtm:trigger:custom event
- https://support.google.com/tagmanager/answer/7679219?hl=en&ref_topic=7679108
- Fire from webpage
- Simple
dataLayer.push({'event': 'button1-click'});- With value of user-defined variable data layer
dataLayer.push({'event':'button1-click','conversionValue':25});
- Refer to google:gtm:user-defined variable:data layer
- Trigger Group
- Only fire after all of the selected triggers have fired at leaswt once
13.12.5. Preview
Click Preview on GTM and normally refresh the webpage and you will get a preview pane just on your browser
13.12.6. UC: Custom Dimension WordPress Post Terms google:gtm:custom dimension
- https://www.competa.com/blog/custom-events-in-google-tag-manager/
- Webpage fires a GTM Custom Event
ga-custom-dimension-post-termswith Custom Variablega-custom-dimension-post-termswhen it's a single post page which has terms - The GTM Event is also a trigger to trigger Tag set up on GTM. The Tag submits a GA Event that sets up Custom Dimension using the Custom Variable
- Refer to ga:gtag:event:custom dimension
13.12.6.1. On website
<head>
<script>
(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({
'gtm.start': new Date().getTime(),event:'gtm.js'});
var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),
dl=l!='dataLayer'?'&l='+l:'';
j.async=true;
j.src='//www.googletagmanager.com/gtm.js?id='+i+dl;
f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-XXXX');
</script>
<!-- End Google Tag Manager -->
<script>
if (typeof dataLayer !== 'undefined') {
// dataLayer is defined after GTM
<?php
$terms = $terms_id = [];
if (is_single()) {
global $post;
$tax = get_taxonomies();
$terms = wp_get_post_terms($post->ID, $tax);
if ( !is_wp_error($terms) && !empty($terms)) {
foreach ($terms as $term) {
$terms_id[] = $term->term_taxonomy_id.'-'.$term->slug;
}
$terms_id[]='';
array_unshift($terms_id, '');
}
//echo '//'.implode( ',', $terms_id );
}
if (!empty($terms_id)) : ?>
dataLayer.push({
'event': 'ga-custom-dimension-post-terms',
'ga-custom-dimension-post-terms': '<?php echo implode( ',', $terms_id ); ?>'});
<?php endif; ?>
}
</script>
</head>
13.12.6.2. Google Analytics
Create Custom Dimension Page Terms, hit type Hit. Say Index number is 1
13.12.6.3. Google Tag Manager
- Create Trigger
My Trigger Name- Type
- Custom Event
- Event name
ga-custom-dimension-post-terms(name does not matter but has to match the code on webpage)- (no term)
- All Custom Events
- (no term)
- The code on website will trigger this event along with a set User-Defined Variable as described below
- Create User-Defined Variable
- Type
- Data Layer Variable
- Name
ga-custom-dimension-post-terms(name does not matter but has to match the code on webpage)- (no term)
- Version 2
- Default value
- blank (which is empty string)
- Create Tag
- Type
- Google Analytics - Universal Analytics
- Track Type
- Event (submit a GA Event)
- Category
- does not matter. Use
ga-custom-dimension-post-terms - Action
- does not matter. Use
ga-custom-dimension-post-terms - Label
- may not matter.. Use custom variable
ga-custom-dimension-post-terms - Enable overriding settings in this tag
- More Settings becomes available
- Custom Dimension
- Index:1, Dimension Value:custom variable
ga-custom-dimension-post-terms
- Tracking ID
- GA-XXX
- Advanced Settings
- Tag firing options
- Once per event
- Triggering
- choose the Custom Event
ga-custom-dimension-post-termspreviously set
13.13. Google Surveys google:surveys
13.14. Google Marketing Platform google:marketing platform
- https://marketingplatform.google.com
- Create an Organization and link products to an organization. e.g.
- Link a GA account with properties
Move a linked product account e.g. GA account to a different organization
- Required permissions
- Source organization: Org admin and Billing admin
- Destination organization: Billing admin
- In Source Organization on Marketing Platform, click 3 dots icon in the org card and click Move accounts from this org to another
- Choose a destination org and choose the accounts you want to move
- Show integrations of the supported products with Google Ads and Search Console
- User Management
- Organization administrators
- Org Admin
- User Admin
- Billing Admin
- Users
- After linking product accounts to the org, any users who have access to that product account are automatically added to the org with the default role of User
- Any Org User who also has Manage Users permissions for other product accounts can add the product accounts to the org
- User Groups
- Policies that govern who has access to the organization
- Organization administrators
- Supported products
- google:ga
- have to individually shared on Data Studio. Email address should be a Google Account
- google:optimize
- google:surveys
- google:gtm
Enterprise includes 2 other DSP products
- DSP
- Demand Side Platform provides advertisers features for buying ad placements online in real time
- Audience targeting
- Advertiser Campaign Management products that can manage ads across display, mobile, social, search and video advertising channels
- Integrate with ad exchanges which serve as a marketplace for ad inventory
- publishers use SSPs to sell ad inventory in ad exchanges
- Such as google:gam
- Some DSPs can integrate directly with SSPs to foster direct buys
- ATD
- Adgency Trading Desk
- ATDs hire software developers, account managers and data analysts who work for the agency's advertising clients. ATDs can use multiple DSPs. Advertisers using the services of an ATD don't get direct access to the available media inventory
- Display & Video 360
- Search Ads 360
13.15. Google SERP Features
13.16. Website Keywords
- Page title
- 65 characters. A list of keywords (<=3) with separators can be used. Order matters.
- (no term)
- Only one
<h1> - (no term)
- How to find keywords to invest
- Keyword Research Tools (KRT)
- find what keywords are within the search engine ranking
- Google Search Console
- see which keywords/queries end up to your site
- Micromoments
- people try to ask small questions for a small duration but several times throughout a period
- I want to know, go, do and buy
- This leads to Buying Cycle
- Early and general
- Research and investigative
- Conversion
- Post-conversion
13.17. Site Audit Tools - check js errors, GA tagging, etc.
13.17.1. ObservePoint
13.18. SEO, Search PPC Strategy Tools and Services
13.18.1. SpyFu.com
Free and no registration is required.
- organic keywords, monthly SE organic clicks
- Google organic clicks vs paid clicks
- paid keywords, monthly PPC clicks, monthly Google Ads budget
- top organic/paid keywords and cost and CPC per paid keyword
- Google Ads history including real ads and corresponding landing pages
- Inbound Links
13.18.2. SERanking.com
Free and no registration is required. Same as SpyFu but it breaks down by country and different SE.
13.18.3. Moz.com
- Moz uses Jumpshot's clickstream data combined with Google AdWords
13.18.3.1. Free tools
- https://moz.com/free-seo-tools
- Keyword Explorer
- free for 20 queries per month
- MozBar
- Chrome extension to show Page Authority, Domain Authority, MozRank, MozTrust
- Online Presence
- check consistency of all your online profiles
- https://moz.com/learn/seo
- Keyword Explorer
- Some other competitors are
- Raven SEO Tools
- SEMrush.com Keyword Magic Tool
- WordTracker
- WordStream
- Ahrefs Keywords Explorer
- https://neilpatel.com/ubersuggest/
- google:ads:keyword-planner
- 5k queries per month
- Metrics
- Difficulty
- invest on high volume keyword with lower difficulty
- Low
- 20-35
- Middle
- 36-50
- Tough
- 51-65
- Very difficult
- 66-80
- After search on Google, Google tells number of results. The more results means more competition
- Keyword Analysis is about to analyze how people use keywords to search content that you provide (e.g. you provide California Vacations)
- Groups
- food? activity?
- (no term)
- Concepts
- Phrase pattern
- people search vacation ideas more than vacation packages
- (no term)
- Prefix
- (no term)
- Suffix
- (no term)
- Synonyms
- Plurals
- people use singular when they want to be specific and plurals when they want general info
- Some other competitors are
- Open Site Explorer (similar to Link Explorer)
13.18.3.2. Pro tools
- On-Page Grader
- Specify a page url and a target keyword, it returns with recommendation
- Link Explorer
- Page Authority, Domain Authority, Linking Domains, Inbound Links, Ranking Keywords, etc.
- Analyze all websites including yours
- Alternatives
- SEMrush
- linkpopularity.com
- checkyourlinkpopularity.com
- majestic.com
- Link-Assistant.com
- RavenTools.com
- SEO-SpyGlass.com
- aHrefs.com
13.18.4. Hype Stat - HypeStat.com
13.18.5. MediaRadar
Estimate unique visitors
13.18.6. SimilarWeb.com
- Similar Web allows you to see GA like web traffic stats. Most of the time estimates are overestimated
13.18.7. Answer the Public AnswerthePublic.com
13.18.8. Keywords Everywhere
13.19. Conversion Rate Optimization - CRO
- https://unbounce.com/landing-page-analyzer/
- https://instapage.com/ https://www.leadpages.net/
- Track phone calls https://www.callrail.com/
- https://www.optimizely.com/, Google Optimize
13.20. Remarketing vs Retargeting
Remarketing is an umbrella term which covers offline, phone, email, social media and websites. Retargeting is a subset of remarketing
Retargeting:
- people have been on your site
- people have been on a competitor site
13.21. Competitive Marketing
- https://www.wordstream.com/blog/ws/2016/05/16/competitive-advertising
- Using Facebook Ad, you can target people with interests in our competitor’s company name
- Use YouTube campaigns, you can target people with searches for our competitor’s videos on YouTube
- Use Gmail campaigns, you can target people with interests in our competitor’s company name and its products. E.g. target people who are interested in Sephora
- Use Display campaigns, you can target people with visits to AND interests in a specific domain. This is new but it is different from remarketing audience. But it’s powerful enough!
- Use Twitter Ads, you can download a competitor’s followers and setup Twitter Ads to target those followers
13.22. Optimize Images
https://moz.com/ugc/10-tips-for-optimizing-your-images-for-search
Google Image sitemaps https://support.google.com/webmasters/answer/178636 Google Image guildlines https://support.google.com/webmasters/answer/114016
13.23. Rich snippet, JSON LD google:json-ld
- LD stands for linked data
- google:search console:structured data markup helper
- https://json-ld.org/
- https://search.google.com/structured-data/testing-tool
- https://developers.google.com/search/docs/guides/prototype
- https://developers.google.com/search/docs/guides/search-gallery
- https://developers.google.com/search/docs/data-types/product
- https://schema.org/docs/full.html
- json shouldn't have double quotes or / backslashes. Any Unicode character should be fine
- Multiple
<script type="application/ld+json">are ok
13.23.1. keywords and syntax tokens
13.23.1.1. @id
The node with @id can be referrenced either internally or externally on internet. The hash indicates Organization object not a page
{
"@context": "http://schema.org",
"@id": "https://www.apple.com/#organization"
"@type": "Organization",
}
13.23.1.2. @graph
Contain an array of nodes that are linked using @id
13.23.2. Product
<script type="application/ld+json"> { "@context" : "http://schema.org", "@type" : "Product", "name" : "Product 1", "image" : "http://mysite.com/uploads/1.jpg" } </script>
Multiple products
<script type="application/ld+json"> [ { "@context" : "http://schema.org", "@type" : "Product", "name" : "Product 1", "image" : "http://mysite.com/uploads/1.jpg" }, { "@context" : "http://schema.org", "@type" : "Product", "name" : "Product 2", "image" : "http://mysite.com/uploads/2.jpg" } ] </script>
13.23.3. Organization, Website google:json-ld:organization
- Refer to wp:plugin:wordpress-seo:json-ld
{
"@context": "http:\/\/schema.org",
"@type": "Organization",
"url": "https:\/\/mywebsite.com\/",
"sameAs": [],
"@id": "#organization",
"name": "My Organization Name",
"logo": "http:\/\/mywebsite.com\/wp-content\/uploads\/2018\/02\/logo.jpg",
"contactPoint": [
{ "@type": "ContactPoint",
"telephone": "+1-401-555-1212",
"contactType": "customer service"
}
]
}
13.23.5. LocalBusiness
https://developers.google.com/search/docs/data-types/local-business https://www.chrisains.com/seo/local-business-schema-mark-via-json-ld/
It should have single address.
<script type="application/ld+json"> { "@context" : "http://schema.org", "@type" : "LocalBusiness", "name" : "Your Business Name", "url" : "http://www.your-domain.co.uk", "logo": "http://www.your-domain.co.uk/images/your-logo.png", "image": "http://www.your-domain.co.uk/images/your-image.png", "address": { "@type" : "PostalAddress", "streetAddress": "1 The High Street", "addressLocality": "Colchester", "addressRegion": "Essex", "addressCountry": "US", "postalCode": "CO1 1AB", "telephone" : "01234 567 891" }, "openingHours": [ "Mo-Fr 09:00-18:00", "Sa 10:00-16:00" ], "priceRange": "$$$", "geo": { "@type": "GeoCoordinates", "latitude": "40.75", "longitude": "73.98" }, "hasmap" : "https://www.google.co.uk/maps/place/Buckingham+Palace/@51.501364,-0.1440787,17z/data=!3m1!4b1!4m5!3m4!1s0x48760520cd5b5eb5:0xa26abf514d902a7!8m2!3d51.501364!4d-0.14189", "sameAs" : [ "https://www.facebook.com/chrisains", "https://twitter.com/chrisains", "https://plus.google.com/+ChrisAinsworth", "https://www.youtube.com/user/chrisains1982" ] } </script>
Multiple LocalBusiness's. This method Google shows no organization…
// Include google:json-ld:organization // LocalBusiness 1 { ..., @id: "https://example.com/url-first-location", ..., "parentOrganization": { "@type": "Organization", "@id": "Same as organization:@id", "name": "Same as organization:name" // Google requires a name for object } }
13.23.6. OpeningHoursSpecification
"openingHoursSpecification": [ { "@type": "OpeningHoursSpecification", "dayOfWeek": [ "Monday", "Tuesday", "Wednesday", "Thursday", "Friday" ], "opens": "09:00", "closes": "21:00" }, { "@type": "OpeningHoursSpecification", "dayOfWeek": [ "Saturday", "Sunday" ], "opens": "10:00", "closes": "23:00" } ]
Open 6pm on Saturday and close on 3am on Sunday
"openingHoursSpecification": { "@type": "OpeningHoursSpecification", "dayOfWeek": "Saturday", "opens": "18:00", "closes": "03:00" }
Open all day on Saturday and close all day on Sunday
"openingHoursSpecification": [ { "@type": "OpeningHoursSpecification", "dayOfWeek": "Saturday", "opens": "00:00", "closes": "23:59" }, { "@type": "OpeningHoursSpecification", "dayOfWeek": "Sunday", "opens": "00:00", "closes": "00:00" } ]
Close for Winter
"openingHoursSpecification": { "@type": "OpeningHoursSpecification", "opens": "00:00", "closes": "00:00", "validFrom": "2015-12-23", "validThrough": "2016-01-05" }
13.23.7. CreativeWork
https://schema.org/CreativeWork
More specific types e.g. Article, Blog, Book, Comment, Game, Map, Movie, Website etc.
13.23.8. VideoObject
{
"@context": "https://schema.org",
"@type": "VideoObject",
"name": "Video Name",
"description": "...",
"thumbnailUrl": "http://...jpg",
"uploadDate": "2018-07-09T13:00:00.000Z",
"duratin": "T1M33S",
"contentUrl": "http://....mp4"
}
13.23.9. BreadcrumbList
13.24. robots.txt
- https://developers.google.com/search/reference/robots_txt
- On Pantheon, a custom robots.txt can be specified but it only works for Live with custom domain. For Pantheon domains, Pantheon always serves its own version
- Valid domains
- Not valid for other subdomains, protocols or port numbers
- Valid for all files in all subdirectories on the same host, protocol and port number
- It has to be in the root folder
- http://example.com/robots.txt">valid for http://example.com/ and
http://example.com/*. Not valid forhttps://example.com,http://example.com:8181/
- Sitemap
- It's require only if you are not the owner and can't submit sitemaps on Google Search Console
- Can specify only one sitemap
- Use full URL with http|https
- Default
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # Disallow: /wp-content/plugins/ # Disallow: /readme.html # Disallow: /refer/ # Allow a useragent # User-agent: PowerMapper # Allow: / # start-of-group # group-member # non-group # All group-member records after a start-of-group record up to the next start-of-group record are treated as a group of records # Only user-agent can start a group # Multiple consecutive start-of-group lines will follow the group-member records following the final start-of-group line # Any group-member records without a preceding start-of-group record are ignored # Only disallow and allow can be a group-member record # sitemap is a non-group record and should be placed at the end # All non-group records are valid independently of all groups Sitemap: http://www.codeinwp.com/post-sitemap.xml
# Pantheon non-Live and non-patheonsite.io domain robots.txt # Pantheon's documentation on robots.txt: https://pantheon.io/docs/bots-and-indexing/ User-agent: * Disallow: / User-agent: RavenCrawler User-agent: rogerbot User-agent: dotbot User-agent: SemrushBot User-agent: SemrushBot-SA User-agent: PowerMapper Allow: /
13.25. Disavow.txt
Submit to https://www.google.com/webmasters/tools/disavow-links
# block a whole domain domain:spamwebsite.com # block individual page url https://www.spamwebsite.com/paid-links/mysite.html
13.26. Design & User Experience
48 pixels width and height for buttons and they should be 32 pixels width and height separated apart.
13.27. hreflang html:hreflang
- https://support.google.com/webmasters/answer/189077
- Google Search Console looks for hreflang to determine language and location. Say if you want to target a generic domain like .com to a specific country/language, you should use hreflang and set in Google Search Console
- For more than one language, use multilingual plugins like wp:plugin:polylang to insert hreflang
- 3 ways to implement
- Inside
<head> - HTTP response header
To indicate a different language version of a URL (e.g. PDF)
Link: <http://es.example.com/document.pdf>; rel="alternate"; hreflang="es", <http://en.example.com/document.pdf>; rel="alternate"; hreflang="en", <http://de.example.com/document.pdf>; rel="alternate"; hreflang="de"
- Sitemap
- ISO 639 and 3166 Country Codes
- Inside
13.27.1. Inside <head>
<!-- General --> <link rel="alternate" hreflang="en-gb" href="http://en-gb.example.com/page.html" /> <link rel="alternate" hreflang="en-us" href="http://en-us.example.com/page.html" /> <link rel="alternate" hreflang="en" href="http://en.example.com/page.html" /> <link rel="alternate" hreflang="de" href="http://de.example.com/page.html" /> <!-- Canada --> <link rel="alternate" hreflang="fr-ca" href="http://www.example.com/fr-ca/about" /> <link rel="alternate" hreflang="en-ca" href="http://www.example.com/en-ca/about" /> <link rel="alternate" hreflang="en" href="http://www.example.com/en/about" /> <link rel="alternate" hreflang="x-default" href="http://www.example.com/" /> <!-- The reserved value hreflang="x-default" is used when no other language/region matches the user's browser setting. This value is optional, but recommended, as a way for you to control the page when no languages match. A good use is to target your site's homepage where there is a clickable map that enables the user to select their country. // wp --> <?php global $wp; $current_url = home_url(add_query_arg(array(), $wp->request)); // echo esc_url($current_url); ?> <link rel="alternate" hreflang="<?php echo get_bloginfo('language'); ?>" href="http://es.example.com/" />
13.27.2. Sitemap
<url> <loc>http://www.domain.com/about/</loc> <xhtml:link rel="alternate" hreflang="en-ca" href="http://www.domain.com/en-ca/about/" /> <xhtml:link rel="alternate" hreflang="fr-ca" href="http://www.domain.com/fr-ca/about/" /> <xhtml:link rel="alternate" hreflang="en" href="http://www.domain.com/en/about/" /> </url>
13.28. Google News
- https://news.google.com
- https://news.google.com/publisher
- Docs
- https://support.google.com/news/publisher-center
- (no term)
Set up content source e.g. website as publisher
- Request inclusion in News Index
- Not necessary but recommended
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"> <url> <loc>http://www.example.org/business/article55.html</loc> <news:news> <news:publication> <news:name>The Example Times</news:name> <news:language>en</news:language> </news:publication> <news:publication_date>2008-12-23</news:publication_date> <news:title>Companies A, B in Merger Talks</news:title> <!-- Optional --> <news:keywords>k1, k2, k3</news:keywords> </news:news> </url> <!-- <url> ... </url> Other news:* elements refer to https://www.google.com/schemas/sitemap-news/0.9/sitemap-news.xsd --> </urlset>
- https://support.google.com/news/publisher-center/answer/74288
- remain in News Index for 30 days once articles are included
- use sitemap index file
- Validate XML
- (no term)
- Assign Labels to Source
- Create sections for a Source
13.29. Privacy Policy
14. UX and Design
14.1. Optimal Workshop
Usability Testing
- Tree testing
- test how well readers can find specific content through navigation
- Card sorting
- how readers would sort or categorize content sections
- First click testing
- what is the first click if readers want specific content?
Ontario Digital Service - Service Design Playbook
- Ethnographic research
- Personas
- Journey maps
- Service blueprints
- Affinity mapping
- Inclusive Design
14.2. Mobile Design
Easy navigation
- CTA button in front and centre position
- Secondary CTA in menu or below the fold
- Keep menu short, always have a button to go home
Easy search
- Search should be one of the first things at the top users see
- Display high relevant search results first
- Provide search filters, preview of # of results, don't show zero results
- Provide previous search
- Ask questions to prefill filters
Easy conversions
- Don't put gate too early which asks for commitment before users actually use the site
- Prefill fields
- Provide CTA for users don't want to fill up complex forms
- Finish converting on another device on another day e.g. save and email jobs to apply for later
Easy forms
- Provider features e.g. number pads, date picker
- Users like tapping toggles instead of text enter or dropdown menu
- Error-check entries immediately
- Multi-part form with progress bar on top
Easy on mobile
- Make product images expandable
- Provide comparison features
- Be responsive with visual feedback after significant actions
- Ask for permissions in context e.g. show overlay over a map to ask for permissions for geolocation
14.3. Google Material Design
Material Design Guidelines :: https://material.io/design/
Material Design Components (MDC)
Codelabs :: https://codelabs.developers.google.com/?cat=Design
14.4. Style Guide
14.5. Pick colors
15. Linux
15.1. Explain Shell
- linux:man
- show the executable path for the current environment
- only searches certain directories but return path and also man page path
15.2. System Version, Distro
uname -a |
all info |
uname -r |
Kernal release |
cat /etc/*release |
Linux distro |
cat /etc/issue |
|
cat /proc/version |
|
lsb_release -a |
Ubuntu only |
uname -aLinux your-server-name 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux-n- network node host name
your-server-name -r- kernal release
4.4.0-116-generic -v- kernal version
#140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 -m- machine hardware name
x86_64 -p- processor type
x86_64 -i- hardware platform
x86_64 - -o
- operating system
GNU/Linux
- Ubuntu only linux:lsb_release
lsb_release -aAll info:No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial
- Ubuntu release e.g. bionic
- Output can be changed by
apt-get dist-upgrade. After running it, it shows16.04.04using kernal4.4.0-116-generic - When Ubuntu is first installed as
16.04.0, kernalgeneric 4.4is installed and kernal4.4.0-xwill always be used when16.04.0is upgraded to16.04.x - It looks like if generic kernal version is installed at the first place, it will stay at that version until special action to replace the generic kernal
- Linux distro
cat /etc/redhat-release
cat /etc/issuecat /proc/version
15.2.1. Debian
cat /etc/debian_version cat /etc/os-release
15.2.2. Upgrade
- sudo apt-get upgrade (-s for dry run)
Install newest versions of all packages currently installed on the system from the sources /etc/apt/sources.list Packages currently installed with new versions will be retrieved and upgraded. No packages are removed or added. e.g. upgrade Firefox use this. New versions of currently installed packages that cannot be upgraded without changing the install status of another package will be left at their current version. An update must be performed first so that apt-get knows that new versions of packages are available.
- sudo apt-get -s dist-upgrade (-s for dry run)
In addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages; apt-get has a "smart" conflict resolution system, and it will attempt to upgrade the most important packages at the expense of less important ones if necessary. So, dist-upgrade command may remove some packages. The /etc/apt/sources.list file contains a list of locations from which to retrieve desired package files. See also apt_preferences(5) for a mechanism for overriding the general settings for individual packages.
By applying all package upgrades, the latest stable kernel will be pulled in.
- sudo apt-get full-upgrade (Ubuntu 14.04+)
full-upgrade performs the function of upgrade but may also remove installed packages if that is required in order to resolve a package conflict.
CentOS :: sudo yum update
Fedora :: sudo dnf update
sudo reboot
15.2.3. Ubuntu upgrade to a newer release
sudo do-release-upgrade
update-manager-core is needed sudo apt-get install update-manager-core
Usually do these 2 first sudo apt-get update, upgrade and dist-grade and then do-release-upgrade
https://www.digitalocean.com/community/tutorials/how-to-upgrade-to-ubuntu-16-04-lts
15.2.4. Kernal Bootloader
By default, the highest versioned kernel will boot.
To see the boot order: grep 'menuentry ' $(find /boot -name "grub.cfg") | cut -f 2 -d "'" | nl -v 0
Default 0 will boot. To change, edit /etc/default/grub
. . . GRUB_DEFAULT=saved GRUB_SAVEDEFAULT=true GRUB_DISABLE_SUBMENU=y . . .
More info: https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel#booting-to-a-non-default-kernel-on-grub-2-distributions Ubuntu 12.04, 14.04 and 16.04 uses Grub 2
15.2.5. Shutdown Ubuntu
shutdown -h now
15.3. Environment and Shell Variables
- Child processes inherit the environmental variables of the parent process
printenvorenvprintenv HOMEorecho $HOMEVARNAME="value"export- Set an env. variable for the current shell and all of its child processes
export TERM=xterm-256color- See a list of env. variables that are manually set
export -p- Set an env. variable using a shell variable
export TEST_VAR- Concatenate
export PATH=/a/new/path:$PATH"
/etc/environment- Set permanently and system wide (all users, all processes) add set variable in
/etc/environmentsudo nano /etc/environment, then logout from current user and login again
- Set permanently and system wide (all users, all processes) add set variable in
env- Modify the environment that programs run in
env VAR1="blahblah" command_to_run command_options
- Modify the environment that programs run in
set(set -o posix; set)- Shell and env. variables should in uppercase while local variables should be in lowercase
TEST_VAR='Hello World!'printenv | grep TEST_VARset | grep TEST_VARexport -n TEST_VARunset TEST_VAR- http://en.tldp.org/LDP/abs/html/internalvariables.html
MACHTYPE- machine type. Used to distinguish OS platforms e.g. apple or linux?
HOSTNAME- system/computer name
HOME- home directory
- (no term)
SHELL- (no term)
BASH_VERSIONSECONDS- number of seconds the Bash session has run. Good for bash script running time
$0- the script file name in which
$0is called PWD- current directory
OLDPWD~-- previous dir
cd -echo ~! _- last argument of previous command executed.
echo $_ls -al >/dev/null && echo $_- last argument is
-al du >/dev/null- last argument is
du
- (no term)
-
export HISTSIZE=10000 # default 500 export HISTFILESIZE=1000000 export HISTTIMEFORMAT='%b %d %I:%M %p ' # using strftime format export HISTCONTROL=ignoreboth # ignoredups:ignorespace # ignorespace means if the command starts with a space, then it won't be saved in history export HISTIGNORE="history:pwd:exit:df:ls:ll"
- delete a line (123), doesn't work for zsh
history -d 123- (no term)
- For Zsh
- Method 1
nano ~/zsh_historyyou will need to restart iterm terminal- Method 2
LC_ALL=C sed -i '' '/porn/d' $HISTFILEandsource ~/.zshrc
15.4. Bash - Bourne-Agin Shell
15.4.1. Config
- Login shell
- A login shell is invoked when
- bash is invoked as in interactive login shell
/bin/bash -i --login - or as a non-interactive shell
/bin/bash --login
- bash is invoked as in interactive login shell
- a login shell is loaded when X11 or virtual terminal is loaded
- e.g. Open Terminal on Mac using Bash
--noprofileoption may be used when the shell is started to inhibit this behavior
Loading sequence
/etc/profile
Ubuntu
- If
$BASHis not/bin/shthen load/etc/bash.bashrc(interactive bash shells) - Load files in
/etc/profile.d
- and First found of the following
- If
~/.bash_profile- Ubuntu doesn't have this file
~/.bash_login- Ubuntu doesn't have this file
~/.profileUbuntu
- If running bash, include
~/.bashrc(interactive non-login shells)
~/.bash_aliases- Add user's private bin to
$PATH ~/bin~/.local/bin
- Add user's private bin to
- If running bash, include
~/.loginfor Mac Bash
- Logout
~/.bash_logout
- A login shell is invoked when
- Subshell
- When an interactive shell that is not a login shell is started
/etc/bash.bashrcand~/.bashrcare run--norcoption may be used to inhibit--rcfileoption will force bash to run a certain file rather than the default 2 files
- Graphical shells, in Ubuntu, will read
/etc/profileand~/.profilebut not for all Linux - In Ubuntu,
.bashrcis loaded when- a login and interactive shell
- Subshell
find \ -type f -iname '*.bashrc'Since
~/.bashrcmay not be included in all Linux distros and all login/subshell instances, may need to manually include it e.g. inside~/.bash_profileif [ -f ~/.bashrc ]; then . ~/.bashrc fi
15.4.2. Prompt, Bash Prompt
- Variables
\$- shows
$if current user is normal or#for root user \u- username
\w- current working dir with tilde
Color
\[\033[COLOR]m\]e.g.PS1="\[\033[31m\]\u@\h:\w$ "- 31,red 32,green 30,black 37,white 33,yellow 35,purple
00mmeans to reset both text style and color- More colors
Text Style
\[\033[ATTRIBUTE; COLORm\]e.g.PS1="\[\033[4;31m\]\u@\h:\w$ "PS1="\[\033[01;04;31m\]\u@\h:\w$ "- normal
- bold for Ubuntu
- dim text
- underline
- blinking
- reverses text and background colors
- hide text
Run Command
PS1="\u@\h on `id -gn` \w\$ "whereid -gnis a command
15.4.2.1. bash-git-prompt
https://github.com/magicmonty/bash-git-prompt
# install on Ubuntu git clone https://github.com/magicmonty/bash-git-prompt.git ~/.bash-git-prompt --depth=1 # add to ~/.bashrc GIT_PROMPT_ONLY_IN_REPO=1 source ~/.bash-git-prompt/gitprompt.sh source ~/.bashrc # see available Ubuntu themes git_prompt_list_themes | grep Ubuntu # change theme in .bashrc GIT_PROMPT_THEME=Solarized # generate a custom theme filed after theme is set to Custom in .bashrc # generate a custom theme based on the Single_line_Ubuntu theme # ~/.git-prompt-colors.sh is created and later you will customize it git_prompt_make_custom_theme Single_line_Ubuntu # in .bashrc GIT_PROMPT_THEME=Custom # toggle gitprompt git_prompt_toggle
15.4.3. Alias
- https://www.digitalocean.com/community/tutorials/an-introduction-to-useful-bash-aliases-and-functions
nano ~/.bashrcalias ll="ls -lhA"
After changing the file
source ~/.bashrcunalias lltype ll
15.4.4. Shortcuts
- https://github.com/fliptheweb/bash-shortcuts-cheat-sheet
- Process
- Exit command line when it's empty
C-d
- Screen
- clear screen
C-l- Stop all output of the currently running foreground command without terminating it
C-s- Resume output of the above
C-q
- Navigation
- Go to command start/end
C-aC-e- Go forward/backward one word
M-f,Esc-f/M-bEsc-b- Go forward/backward one character
C-fC-b. As bash:tmux usesC-b, doubleC-bto achieve the same- Move to the beginning and hit again to go back the original cursor position
C-xx
- Delete or cut (add to clipboard)
- Delete or cut from the beginning to current cursor
C-u- Delete from the current cursor to the end
M-d- Cut from the current cursor to the end
C-k- Cut the word before the cursor
C-w- Delete one character at cursor
C-d- Delete one character before cursor
C-h
- Paste
- yank
C-y
- Undo
- Undo last key press
C-_
- Swap
- Swap the current word with the previous word
M-t- Swap the last 2 characters order
C-t
- Capitalizing
- Capitalize from cursor to the end of the current word
M-u- Uncapitalize from cursor to the end of the current word
M-l- Capitalize the character at cursor, but cursor will move to the end of the current word
M-c
- History
- Go to previous command
UporC-p- Go to next command
DownorC-n- Revert to a command you've pulled from history if it's edited
M-r- Recall the last command by searching
C-r- Run a command you found with
C-r C-o- (no term)
- Leave
C-rwithout running a command - Search forward
C-r- Trick
- add a comment and later search that comment
- e.g. run
ls -al #longcommenttoIDthiscommand C-rand type#longcommenttoIDthiscommand
- Run a command you found with
- Return last run command without running it
echo !!- Run last run command
!!- run 2nd last command
!-2- run command at a number
!123
- Return last run command starting with cat and put this command to the last run command history
!cat:p- Use last run command's arguments
mkdir /new/paththencd !$- Modify last run command by replacing the first instance of nanp with nano
nanp .ssh/configthen^nanp^nano- Run the 455th command in history
!455- (no term)
- Run a command with several arguments that have minor difference using curly brackets
{}mv /path/to/file.{txt,xml}mkdir myfolder{1,2,3}cp /etc/rc.conf{,-old}eq. tocp /etc/rc.conf /etc/rc.conf-oldmv /etc/rc.conf{-old,}eq. tomv /etc/rc.conf-old /etc/rc.conf
15.4.5. Here Documents
# EOF is not quoted means parameter expansion and command substitution and arithmetic expansion is possible # \ must be used to quote the characters \, $ and ` mysql -u root -p mydb <<EOF SELECT * FROM abc WHERE name LIKE '%lili%' EOF # use <<-EOF or <<-'EOF' to strip tab characters in the actual input # EOF is quoted, no expansion mysql -u root -p mydb <<'EOF' SELECT * FROM abc WHERE name LIKE '%lili%' EOF
15.4.6. Options
# List of all options and their on/off status shopt # See if an option is enabled or not shopt extglob # List of all enabled options shopt -s # List of all unset/disabled options shopt -u # Enable an disabled option shopt -s histappend # Disable all options shopt -u # Eanble a disabled option shopt -u cmdhist # Enable extglob for just the running command: prints all immediate child directories and files whose name is not foo bash -c -O extglob 'echo !(foo)'
15.5. zsh
15.5.1. Oh My Zsh
Basics
# install script from https://github.com/ohmyzsh/ohmyzsh sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" zsh --version # .zshrc is created or modified # manually upgrade zsh upgrade_oh_my_zsh # update changes source ~/.zhrc brew install zsh-autosuggestions
- Inside
~/.zhrcZSHdefault folder:~/.oh-my-zsh- This folder is set as ZSH folder
- can be used to change the default folder:
~/path/to/custom - Sub directories
lib- Oh My Zsh config files broken down by module e.g.
history.zshandsource $ZSH_CUSTOM/lib/history.shis run - Don't change these files
- Oh My Zsh config files broken down by module e.g.
pluginscustom- Create any
*.zshdirectly in the folder e.g.~/.oh-my-zsh/custom/alias.zsh
- Create any
themes
source $ZSH/oh-my-zsh.sh- Initialize Oh My Zsh
- Followed by your own custom settings
- https://www.freecodecamp.org/news/how-to-configure-your-macos-terminal-with-zsh-like-a-pro-c0ab3f3c1156/
15.5.1.1. Built-in plugins
- copyfile
- copyfile path/to/file to copy the file into clipboard
- (no term)
- macos
- ofd
- open the current directory in a Finder window
- pfd
- return the path of the frontmost Finder window
- cdf
- cd to the current Finder directory
- (no term)
jsontools
# pretty print echo '{"b":2, "a":1}' | pp_json # echo two separate json objects and pretty print both echo '{"a": "b"}\n{"c": [1,2,3]}' | pp_ndjson
15.5.1.2. Command alias
d- eq.
dirs -vlist last visited directoriescd -1- go to directory number 1
cd -2
-cdto last visited directorycdor~cd ~...cd ../......cd ../../.......cd ../../../../cd /mdmkdir -pmkcd- create a new directory and change to it
take URLortake path/to/directory- like
mkcdand it handles remote URLs. Download the file and extract it and go to the newly extracted/downloaded/cloned directory ./+ tab + tab- then use arrow keys to view and select a file or directory under the current directory
15.5.2. iTerm2
15.5.2.1. Install
# install or upgrade CLI developer tools for Xcode xcode-select --install # To get the current Xcode version (when only CommandLineTools is installed) pkgutil --pkg-info=com.apple.pkg.CLTools_Executables | grep version # if an error, run this to reset `xcode-select` # xcode-select -r # install homebrew brew install --cask iterm2 # download Color Presets cd Downloads curl -O https://raw.githubusercontent.com/MartinSeeler/iterm2-material-design/master/material-design-colors.itermcolors # Go to iTerm2 > Preferences > Profiles > Colors Tab # Click Color Presets… at the bottom right # Click Import… # Select the material-design-colors.itermcolors file # Select the material-design-colors from Load Presets
15.5.2.2. Settings ⌘ + ,
- Profiles
- General
- Working Directory
- Reuse previous session's directory
- Advanced
- Visual bell flashes the whole screen, not just a bell icon
- General
15.5.2.3. Shortcuts
- Tab
- new tab
Cmd-t- close current tab
Cmd-w- Switch to a tab left or right
Cmd-S-[Cmd-S-]- Switch to a tab
Cmd-1,Cmd-2- Edit Session
⌘ i- Tab title
- Change tab name
- Pane
- new vertical pane
Cmd-d- new horizontal pane
Cmd-S-d- Close current pane
C-d- Switch to a pane left or right
Cmd-[Cmd-]
⌘ ;- Preference > Profiles > Keys > Key Mappings
- Use preset Natural Text Editing
- Add
M-basEsc-bM-fasEsc-f
- Add
- Use preset Natural Text Editing
15.5.2.4. Recent Directories ⌘ ⌥ /
15.5.2.5. Terminal Replay :: ⌥ ⌘ b and then use arrow keys and hold ⇧ plus arrows to replay faster
15.5.2.6. Show execution time ⇧ ⌘ e
15.5.3. .zshrc
- get some ideas from `cat /etc/zshrc`
15.5.3.1. Customize Prompt
man zshmisc # Scroll to section SIMPLE PROMPT ESCAPES to get all variables / Escapes which can be used in Prompt nano ~/.zshrc PROMPT="%(?:%{$fg_bold[green]%}➜ :%{$fg_bold[red]%}➜ )" PROMPT+=' %{$fg[cyan]%}%~%{$reset_color%} $(git_prompt_info)' source ~/.zshrc
15.5.4. .zsh_history
- Delete a command from history
15.5.5. Options
- See Bash -> options
# All enabled options setopt # All disabled options unsetopt
15.6. User Management, Superuser, sudo
15.6.1. Create and delete a user, Add user to sudo group
sudo adduser coreysudo deluser corey- Use
useraddfor non-interactive (Docker)- Example
useradd --user-group --create-home newusername
Options
--usergroup, -U- Create a group with the same name as the user
--create-home, -m- Create home directory if does not exist
--shell, -s- Assign a default shell for this user.
/bin/falsereturns false then exit (logout)
useradd --user-group --create-home --shell /bin/false newusername-o -u 1000- change UID to 1000, -o is to allow duplicate UID
- Example
15.6.1.1. Add user to the sudo group
- See 15.6.2 to fine tune permissions
As root user, run this to add the user to the sudo group
gpasswd -a username sudo # or usermod -aG sudo username # -aG keeps the existing groups the user belongs to and add the user to another group # list groups a user is in groups username # list groups for the current user id -nG # or groups sudo usermod -aG agroupname ausername
15.6.2. Sudoer file
- Run
sudo visudoto see the sudoers file to check:- permissions given to each group
- check sudo environment variable setting
- Instead of directly editing using
sudo visudo(don't directly edit file/etc/sudoers), add files to/etc/sudoers.d - Examples
root ALL=(ALL:ALL) ALL- User root can run sudo as all hosts (1st ALL), all users, all groups and apply to all commands
%sudo ALL=(ALL:ALL) ALL- Start with
%means is a group lili2 ALL=(ALL:ALL) ALL- lili2 can run any command or login as any user
ubuntu ALL=(ALL) NOPASSWD:ALL- ubuntu can run any command without using password on any host or login as any user
groups username- More info
15.6.3. Change a user's password
- Login as root and run
passwdto change root password - change another user's password
15.6.4. List all users cut -d: -f1 /etc/passwd or cat /etc/passwd linux:/etc/passwd
- Login username
- Password (always x)
- UID (<500 are system accounts, > 500 are users)
- Group ID (GID)
- Comment field
- Location of HOME directory
- Default shell for the user
Refer to bash:awk
awk -F':' 'BEGIN{OFS="\t"; print "Login Username","Password","UID", "Group ID (GID)", "Comment", "Home Dir", "Default shell"} {print $1,$2,$3,$4,$5,$6,$7}' /etc/passwd | column -t -s $'\t'
15.6.5. Sudo
- Options
-S, --stdin- linux:sudo:password
-H- use the target user's home directory
-n, --non-interactive- avoid prompting the user for input of any kind. If a password is required, it will stdout an error message and exit
15.6.5.1. Switch to another user
- Switch to superuser/root
- Debian
sudo su- RedHat
sudo -
- Login to another user, Run command as another user
- Login (need to type password)
/bin/su - usernameorsu liliadminexitto go back to original user
- Run as a user (need to type password)
sudo -u lynda whoami/bin/su - username -c 'whoami'
As root user, run as another user without typing password
sudo -H -u username bash -c 'whoami'
- See linux:sudo:o:H
- Login (need to type password)
15.6.5.2. Sudo run multiple commands
sudo bash -c 'apt-get update;apt-get install'
15.6.5.4. Require no password when sudo is run as another user
- User abc has sudo privilege. When abc is signed in and run a script/command that requires sudo, system will ask for password
- e.g.
sudo ./path/to/script.shThis bash file can containmake startwhere recipes may not have sudo in front - bypass so password input is not required
- e.g.
Instead of changing
/etc/sudoersfile directly, create/edit a file/etc/sudoers.d/mybypass# See if there are some sample sudo cat /etc/sudoers sudo ls -al /etc/sudoers.d/ # found one sample sudo cat /etc/sudoers.d/mybypass # have to use visudo to edit /etc/sudoers and /etc/sudoers.d/anyfile files. # -f is to specify a file, otherwise is the default /etc/sudoers file sudo visudo -f /etc/sudoers.d/mybypass # Add this line abc ALL = (root) NOPASSWD: /path/to/script.sh # allow user abc on ALL hosts to run a specific command without entering password # all other sudo commands will still require a password # ~/etc/sudoers.d/mybypass file should have permission 0440 and owner:group is root:root
15.7. SSH, scp, sshpass
15.7.1. Install SSH Server
Ubuntu
sudo apt install ssh # should be started service ssh status sudo service ssh stop sudo service ssh start # disable ssh service sudo systemctl disable ssh sudo systemctl enable ssh
15.7.2. Create .ssh folder
cd ~
mkdir .ssh
chmod 700 ~/.ssh && chmod 600 ~/.ssh/*
15.7.3. Generate client public and private keys, transfer public key to remote
- Generate a private key on local machine
ssh-keygen -t rsa -b 2048-t rsa- type of key
-b 2048- bits
- (no term)
-f myfilenamemyfilenamefile- private key
myfilename.pubfile- public key
- (no term)
-P existingPassPhrase-p- request changing the passphrase of a private key file instead of creating a new private key
- Under folder
/Users/username/.ssh/id_rsa.pub is the public key and id_rsa is the private key cat ~/.ssh/id_rsa.pub- When using private key you might encounter
Permissions 0777 for '~/.ssh/id_rsa_your_private_key' are too open
Use chmod 400 ~/.ssh/id_rsa_your_private_key to fix it chmod:400
- Move public key to remote
- Fast way
- Append the client public key to remote host's
/.ssh/authorized_keys ssh-copy-id remoteusername@123.45.56.78- Or specify a public key file
ssh-copy-id -i ~/.ssh/id_rsa.pub remoteusername@123.45.56.78
- Append the client public key to remote host's
- Fast way
- Show fingerprint of specified public key. Private RSA1 keys are also supported
ssh-keygen -lf publickey
15.7.3.1. Manual way move pub key to remote
- On the remote server, login as a normal user
sudo su - username - Create
~/.sshdirectorymkdir .ssh && chmod 700 .ssh - Create file and paste the public key
nano .ssh/authorized_keys
Or concatenate it onto authorized_keys file manually
# backup first cp authorized_keys authorized_keys_Backup cat id_rsa.pub >> authorized_keys
Change permission chmod 600 .ssh/authorized_keys chmod:600
Add a name to id_rsa.pub
cat yourpubkey.pub ssh-rsa longstring yourname@yourhost
15.7.3.2. Move private key to another computer
mkdir .sshchmod 700 .sshcd .ssh && nano yourprivate_keychmod 400 yourprivate_key
15.7.3.3. Remove passphrase for private key id_rsa
cd ~/.ssh ssh-keygen -p -f id_rsa # enter old passphrase and new
The following no longer works
# Make a backup cp ~/.ssh/id_rsa ~/.ssh/id_rsa.backup rm ~/.ssh/id_rsa # remove passphrase openssl rsa -in ~/.ssh/id_rsa -out ~/.ssh/id_rsa_new # enter old passphrase cp ~/.ssh/id_rsa_new ~/.ssh/id_rsa
15.7.4. github:ssh:key:fingerprint
15.7.5. Config linux:ssh:config
Create ssh host alias and the private key to use in ~/.ssh/config file
cd ~/.ssh && nano config
chmod 600 ~/.ssh/config
Host *
ServerAliveInterval 240
ServerAliveCountMax 2
Host scotch
HostName scotch.io
User nick
Host example2
HostName example.com
User root
Password abc
Host example3
HostName 64.233.160.0
User userxyz123
Port 56000
ServerAliveInterval 240
ServerAliveCountMax 2
Host example4-Switch-to-root
HostName 64.233.160.0
User nonRootUserThatIsSudoerCanSwitchToUser
# This solution still doesn't provide the root user for SFTP through SSH.
# SFTP still uses the non-root user
# But the solution will change the root once logged in
RemoteCommand sudo su -
# optional. Required if remote requires TTY
RequestTTY yes
Host amazon1
HostName ec2.amazon.com
User ec2-user
IdentityFile /path/to/special/privatekey/amazon.pem
ServerAliveInterval 240- Sends a packet to the server every 240 seconds (4 minutes). If the client does not receive a response after 2 tries, it closes the connection.
- In Putty, under Connection > Seconds between keepalives > 240
- To kill the connection when it gets hang up,
~.
- (no term)
- Convert Putty private key .ppk to OpenSSH using puttykeygen
- Load > Conversions (in top menu): Export OpenSSH key, save as
id_rsa_yourwebwithout file extension
- Load > Conversions (in top menu): Export OpenSSH key, save as
15.7.6. Remove a known_host
- TS: SSH Client, Warning Remote Host Identification Has Changed!
ssh-keygen -f "/home/vagrant/.ssh/known_hosts" -R "[172.22.0.3]:2222"ssh-keygen -f "/home/vagrant/.ssh/known_hosts" -R "[172.22.0.3]:0"use 0 for all ports
15.7.7. Use private key from SSH server to connect
AWS uses .pem file as the private key on its SSH server. Convert this .pem file in PuttyGen and later uses it to connect to AWS Refer to aws:ssh
15.7.8. Restart SSH
- Ubuntu
- See 15.7.9.1 (22.10)
sudo systemctl reload sshd(16.04)sudo systemctl restart sshd(16.04)sudo restart ssh(14.04)
service ssh restart
15.7.9. sshd_config SSH Daemon Config File
15.7.9.1. Socket-activated
- Since Ubuntu 22.10, SSHd uses socket-based activation
- Some settings on sshd_config is not used any more
- Add port
- Add port 443 on top of the default port 22
mkdir -p /etc/systemd/system/ssh.socket.d cat >/etc/systemd/system/ssh.socket.d/listen.conf <<EOF [Socket] ListenStream= ListenStream=443 EOF sudo systemctl daemon-reload sudo systemctl restart ssh # Should list `Server listening on :: port 22` and 443 systemctl status ssh
15.7.9.2. Backup
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.factory-defaults sudo chmod a-w /etc/ssh/sshd_config.factory-defaults
15.7.10. Disallow SSH as root user, SSH Daemon Config linux:ssh:disallow_root
15.7.11. Disable Password Authentication
- Force to use public key only instead of plain password only
sudo nano /etc/ssh/sshd_config- (no term)
- Replace
#PasswordAuthentication yeswithPasswordAuthentication no - (no term)
- linux:ssh:restart
- (no term)
These are supposed to be default. If not, change to as below
PubkeyAuthentication yes ChallengeResponseAuthentication no
15.7.12. Login without setting public keys bash:sshpass
https://www.cyberciti.biz/faq/noninteractive-shell-script-ssh-password-provider/
apt-get install sshpass sshpass -p 't@uyM59bQ' ssh -o StrictHostKeyChecking=no username@server.example.com # StrictHostKeyChecking is for shell script to ignore prompt # Use a file for password, this is the most reliable way when it has special characters for rsync echo 'myPassword' > myfile chmod 0400 myfile sshpass -f myfile ssh vivek@server42.cyberciti.biz # scp sudo sshpass -p 'abc' scp -o StrictHostKeyChecking=no -r /local/path/archive/ user@1.2.3.4:/remote/path/dtp-tmp/ # use with rsync rsync --rsh="sshpass -p myPassword ssh -l username" server.example.com:/var/www/html/ /backup/ # ssh -l means to specify a username to login # use environment variable SSHPASS to store password and it has to SSHPASS SSHPASS='yourPasswordHere' rsync --rsh="sshpass -e ssh -l username" server.example.com:/var/www/html/ /backup/ # gpg encrypted file echo 'mySshPasswordHere' > .sshpassword gpg -c .sshpassword rm .sshpassword gpg -d -q .sshpassword.gpg > fifo; sshpass -f fifo ssh vivek@server1.cyberciti.biz
15.7.13. Random password
# base64 will likely to have = at the end, remove those to have length 16 openssl rand -base64 17 # in the middle there may be + or / characters, try the following to remove those. Ensure the first number 20 is larger than the last number which is the final length openssl rand -base64 20 | tr -d "=+/" | cut -c1-16 # hex has only number and lowercased letters openssl rand -hex 16
15.7.14. Run command
# string does not need to be escaped dbpw='$@$$$' filedate=$(date +%FT%H%M%S) ssh user1@server1 command1 ssh user1@server1 'command2' ssh user1@server1 'command1 | command2' ssh usehostinssh-config 'long command' # escape single quote inside command # wrong ssh host1 'mysqldump -u user --password='$(dbpw)' dbname > ~/db.sql' # correct: ssh host1 'mysqldump -u user --password='"'"'$(dbpw)'"'"' dbname > ~/db.sql' # '"'"' :: # ' end the first quotation # " start second quotation # ' quoted character # " end second quotation using double-quotes # ' start third quotation # result # mysqldump -u user --password='abc' dbname > ~/db.sql # You might not want to force to use single quote ssh host1 "mysqldump -u user --password='$dbpw' dbname > ~/db-$filedate.sql" # Save dump to local ssh host1 "mysqldump -u user --password='$dbpw' dbname" > ~/db-live-"$filedate".sql
15.7.15. SSH tunneling
- TCP Port forwarding is also called tunneling
- Say your IMAP port 143 to gmail is blocked by Wi-fi firewall
- Default ssh port is 22. Default https port is 443
For Github and BitBucket, you can directly change to use port 443 without doing anything
Host github.com Hostname ssh.github.com Port 443 Host bitbucket.org Hostname altssh.bitbucket.org Port 443
- If you have control of the target server's SSH server, simply just add
Port 443to/etc/ssh/sshd_configand use-p 443locally- See 15.7.9.1 for Ubuntu 22.10+ to add port 443
- If you don't have control of the target server's SSH server
- Assume you have a Host entry for intermediate host
myserverusing port443- (no term)
- In your intermediate host
my.server.com, addPort 443as a second SSH port and setAllowTcpForwarding yesin/etc/ssh/sshd_config, 15.7.8
On your local set another Host entry in
~/.ssh/configHost targetserver-localhost Hostname 127.0.0.1 Port 10143 User myyser IdentityFile ~/.ssh/id_rsa_targetserver RemoteCommand sudo su - RequestTTY yes Host targetserver Hostname target.server.ip.0 User myuser IdentityFile ~/.ssh/id_rsa_targetserver RemoteCommand sudo su - RequestTTY yes- And initiate the local port forwarding on local
ssh -L 10143:target.server.ip.0:22 myserver- Any traffic to localhost:10143 is forwarded to
target.server.ip.0at port 22 throughmyserverat port 443
- Any traffic to localhost:10143 is forwarded to
- (no term)
- Now you can
ssh targetserver-localhoston local
15.7.15.1. Local Port Forwarding
- We are going to forward local client port 10143 to an intermediate host
my.server.comat port 443, which forwards to mail.google.com at 143- In the intermediate host with ssh server installed, add the line
Port 443to /etc/ssh/sshd_config, linux:ssh:restart,sudo ufw allow 443/tcp - Check if it connects from local
ssh -p 443 my.server.com Then open, listen and forward local client port 10143 to destination imap.google.com at port 143 using intermediate host at 443
ssh -L localhost:10143:mail.google.com:143 -p 443 user@my.server.com
- Change local email client setting to use localhost at port 10143
- In the intermediate host with ssh server installed, add the line
- See network:port on port range you should use
- See if port 10143 is being used on localhost
lsof -i :10143
Now SSH to localhost:10022 will be forwarded to target_server_user@target.server.com:22 through SSH server user@my.server.com:443
ssh -p 10022 target_server_user@localhost scp -P 10022 localhost:/target/server/path/ /local/path/ rsync -av -e "ssh -p 10022" /target/server/path localhost:/local/path/- Local forwarding. specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side. Allocate a socket to listen to
porton the local side, optionally bound to the specifiedbind_address. Whenever a connection is made to this port, the connection is forwarded over the secure channel, and a connection is made tohostporthostportfrom the remote machine.- Summary
- [localhost | bind_address]:port > -p 443 user@my.server.com (SSH) > host:hostport (access from my.server.com)
- (no term)
Multiple redirections can be set up in the same tunnel
# add -f to go into background and -N to not launch a shell ssh -L localhost:10143:mail.google.com:143 localhost:10022:target.server.com:22 -p 443 user@my.server.com
-D [bind_addres:]port- Similar to -L but the destionation
host:hostportis not specified. It's called dynamic application-level port forwarding. - e.g.
ssh -CND 10022 -p 443 user@my.server.com- Open port 10022 on localhost, any traffic going into this port is forwarded to
user@my.server.comat443, and finally to any IP of any port on the internet - setup SSH to my.server.com as usual and then Connection > SSH > Tunnels > Uncheck everything except adding source port e.g. 10022 so that Forwarded ports become D10022 and check Dynamic and Auto.
- Open port 10022 on localhost, any traffic going into this port is forwarded to
- Now a SOCKS proxy server is set up. SOCKS is a protocol
- 127.0.0.1:10022
- localhost:10022
- Set up application e.g. Firefox, Chrome to use the SOCKS 5 tunnel
- Firefox
- https://www.digitalocean.com/community/tutorials/how-to-route-web-traffic-securely-without-a-vpn-using-a-socks-tunnel
- Firefox supports
Proxy DNS when using SOCKS v5. If the app does not support DNS proxy, ISP can see traffic send to the DNS server specified in the app.- You may need to specify each domain-IP lookup manually. On my.server.com, set up custom records to resolve domains in file
/etc/hosts
- You may need to specify each domain-IP lookup manually. On my.server.com, set up custom records to resolve domains in file
Chrome
"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" ^ --user-data-dir="%USERPROFILE%\my-proxy-profile" ^ --proxy-server="socks5://127.0.0.1:10022" # or chrome.exe /usr/bin/google-chrome \ --user-data-dir="$HOME/my-proxy-profile" \ --proxy-server="socks5://127.0.0.1:10022" "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \ --user-data-dir="$HOME/my-proxy-profile" \ --proxy-server="socks5://127.0.0.1:10022" # --user-data-dir :: user profile will be created automatically if not exists # The hostname for destination URLs will be resolved by the proxy server. # ftp:// URLs though a SOCKS proxy is not implemented # To prevent Chrome from sending any DNS requests over the network (e.g. DNS prefetch uses direct DNS resolve even though proxy is used) # --host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE myproxy" # Verify proxy setting # chrome://net-internals/#proxy # Verify DNS resolving chrome://net-internals/#dns # Verify proxy logic for individual requests # chrome://net-internals/#events # Use HTTP proxy "foopy:80" for http URLs and HTTP proxy "foopy2:80" for ftp URLs # Without scheme and a single proxy :: use that proxy for all URLs chrome.exe --proxy-server="http=foopy:80;ftp=foopy2" # --proxy-bypass-list=(<trailing_domain>|<ip-address>)[:<port>][;...] # * can be used # --proxy-server="foopy:8080" --proxy-bypass-list="*.google.com;*foo.com;127.0.0.1:8080"
Git
git config --global http.proxy 'socks5://127.0.0.1:10022' git config --global https.proxy 'socks5://127.0.0.1:10022'
- Firefox
- For applications that don't support using SOCKS
Use
torifyorusewithtor(installed along with tor) as shell wrapper to re-direct command line traffic using SOCKS https://neverendingsecurity.wordpress.com/2015/03/14/proxify-applications-with-torsocks-tsocks-and-proxychains-on-linux/sudo apt-get install tor # re-direct for `telnet google.com 80` torify telnet google.com 80 # or usewithtor telnet google.com 80 # On top of wrapping command line, some applications/software are supported userwithtor pidgin
tsockssudo apt-get install tsocks nano /etc/tsocks.conf # server = 127.0.0.1 # server_port = 100220 # Run applications using tsocks. Run locally (applications, command line e.g. ~ftp~ ~ssh~, you don't need to open different SSH tunnels/ports!): tsocks skype tsocks kopete tsocks kmail tsocks ftp someserver.com tsocks ssh user@target.server.com # Notes # - FTP has to be in passive mode # - Some applications might not support tsocks
- sshuttle
- Similar to -L but the destionation
- Other options
-N- Don't run any command on
user@my.server.comafter the connection is first established -ffork the process to the background
# To terminate ps aux | grep ssh # Find the PID by searching the command kill $thePID
-C- compress the data before sending
-q- quiet mode
15.7.15.2. Remote Port Forwarding
-R [bind_address:]port:host:hostport user@my.server.com- Remote port forwarding (on local, initiate remote port fording for my.server.com). Specifies that the given port on the remote (server) host is to be forwarded to the given host and port on the local side. This works by allocating a socket to listen to
porton the remote side, and whenever a connection is made to this port, the connection is forwarded over the secure channel, and a connection is made tohostporthostportfrom the local machine. - [bind_address | empty means any IP] (access from internet) >
portat my.server.com > -p 443 user@my.server.com (SSH) > host:hostport (usually host is localhost) - host:hostport is usually localhost:8888
- e.g. traffic from the internet access my.server.com:9000 will be forwarded to your localhost:8001. Run this locally, and the SSH server my.server.com will open a port 9000 which forwards request to your localhost:8001
ssh -R 9000:localhost:8001 -p 443 user@my.server.com- linux:ssh:restart">
GatewayPorts yesGatewayPorts yes- my.server.com allows anyone to connect to port 9000. SSH server my.server.com will open a port 9000 for anyone
- Anyone
- If the server is on the public internet, anyone on the Internet can connect to port 9000
GatewayPorts clientspecified- e.g. only connections from the IP address 1.2.3.4 to port 9000 is allowed
ssh -R 1.2.3.4:9000:localhost:8001 -p 443 user@my.server.com
- Remote port forwarding (on local, initiate remote port fording for my.server.com). Specifies that the given port on the remote (server) host is to be forwarded to the given host and port on the local side. This works by allocating a socket to listen to
- allows remote hosts to connect to local forwarded ports.
- don't run command. Just forwarding ports.
15.7.15.3. SOCKS on Windows
Windows Internet Options can set SOCKS5 as proxy. However, DNS will be resolved locally at client. While Chrome and FireFox support resolving DNS on the proxy DNS.
On Windows, use the portable SocksCap64, run it as admin, setup SOCKS5 proxy, and add individual Windows application as executable, select the app and click Run!
15.7.16. TS - WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
- The unique fingerprint of the destination server does not match what was stored in the client SSH
known_hosts. Common reasons on destination server- root password is changed
- It is rebuilt
- Same IP address for multiple remote systems
- delete the entries corresponding to the destination server's IP in
known_hosts- Linux and MacOS
.ssh/known_hosts
15.8. fail2ban linux:fail2ban
Ban access to services (e.g. SSH) for max tries and ban time based on service's log.
Check if it's installed ll /etc/fail2ban
Install sudo apt-get update sudo apt-get install fail2ban
Create jail.local based on jail.conf by commenting out every line
awk '{ printf "# "; print; }' /etc/fail2ban/jail.conf | sudo tee /etc/fail2ban/jail.local
Modify jail.local sudo nano /etc/fail2ban/jail.local
By default, fail2ban is enabled for sshd service only and disabled for every service (allow every access)
[DEFAULT] ignoreip = 127.0.0.1/8 # 600 seconds = 10 minutes bantime = 600 maxretry = 3 # 3 tries within 10 minutes findtime = 600 # For a service, fail2ban can search if a log file has certain text and mark it as a authentication failure [nginx-http-auth] enabled = true # regex pattern is defined in /etc/fail2ban/filter.d/ngix-http-auth.conf # You usually don't modify these .conf files
- Restart the service
sudo service fail2ban restart- (no term)
- See which service has fail2ban enabled :;
sudo fail2ban-client status - See detail about a service that has fail2ban enabled
sudo fail2ban-client status sshd- Unban an IP for service
sudo fail2ban-client set apache unbanip 111.111.111.111- (no term)
- More info
- (no term)
- Apache and fail2ban
15.9. System
- Resource Limits
sudoedit /etc/security/limits.conf- (no term)
System Logs
/var/log/syslog general system log /var/log/auth.log system auth logs /var/log/mail.log system mail logs /var/log/messages general log messages /var/log/dmesg kernel ring buffer msg, e.g. system bootup - Search a word `warthog` in .gz files
zgrep -i warthog /var/log/*.gz- (no term)
uptimewho- who are using the system right now
15.10. Filesystem
15.10.1. /dev /mnt /media
/dev/sd*1- sd for SATA or SCSI drive, numeric 1 for partition, * for a, b, etc.
/dev/hd*1- hd for IDE drive, * for a, b, etc.
- (no term)
/media/win1- virtual directory where files are accessed file here.
- Equivalent to
d:\.One or more partitions can be assigned to one media
sudo mkdir /media/win2- create a virtual directory is easy
sudo mkdir /mnt/my_mount- also creates a virtual directory
sudo mount -t ext4 /dev/sda1 /mnt/my_mount- Mount a partitioned device to a virtual directory
sudo gedit /etc/fstab
, /dev/hdb1 media/win1 vfat users,rw,owner,umask=000 0 0
, then reload sudo mount -a
- Permanently mount during bootime
15.10.2. bindfs linux:bindfs
- Tested on Ubuntu Xenial
- Create a user
devoneand mount/home/devone/websites/application1to/var/www/application1which is owned bywww-data devonecan create/edit files while the destination folder maintain the same ownerwww-dataand permissions
apt-get update apt-get -y install bindfs sudo adduser devone mkdir -p /home/devone/websites/application1 chown -Rf devone:devone /home/devone/websites chmod -Rf 770 /home/devone/websites nano /etc/fstab bindfs#/var/www/application1 /home/devone/websites/application1 fuse force-user=devone,force-group=devone,create-for-user=www-data,create-for-group=www-data,create-with-perms=0644,chgrp-ignore,chown-ignore,chmod-ignore 0 0 mount /home/devone/websites/application1 # if your system yells about force-user or force-group not being defined: # replace force-user by owner # replace force-group by group # Test su - devone cd ~/websites/application1 touch hello.txt ls -al hello.txt # -rwxrwx--- 1 devone devone 0 sept. 10 17:15 hello.txt exit cd /var/www/application1 ls -al hello.txt # -rwxrwx--- 1 www-data www-data 0 sept. 10 17:15 helloworld.txt
Make sure www-data user owns /var/www can prevent devone directly access the folder
chown www-data:www-data /var/www chmod 770 /var/www
If /etc/fstab is modified and you want to remount to reflect the changes. Unmount the existing first
umount /home/devone/websites/application1 # if that fails, try to navigate out of the /home/devone/websites/application1 for all users # or use -l for lazy and -f for force # umount -l /home/devone/websites/application1 mount /home/devone/websites/application1 # or mount -a to mount all entries in /etc/fstabs
http://blog.netgusto.com/solving-web-file-permissions-problem-once-and-for-all/
15.10.3. /etc system configuration files
15.10.4. Disk space
df -h |
Disk free space on OS |
| h for human-readable | |
du -sh |
Current folder size. -s shows total only |
du -sh ./* |
List sizes of 1st level child folders and files |
du -sh ./wp-content/* |
List sizes of 1st level child folders of one folder |
- Disk usage in current folder including all sub folders but excluding one sub folder
du --exclude=./wp-content/uploads --exclude=./wp-content/error_log -sh- Exclude a file?
du --exclude-from 'path/to/files.txt'- (no term)
--max-depth# Only get down to 1 level. When max-depth > 0, -s cannot be used du --max-depth=1 -h # sort by size in MB du --max-depth=1 -h -m | sort -h # Do not count subdirectories size du --max-depth=1 -Sh
- (no term)
Search which directory has taken up space, Highlight lines that have M or G in front for file size
du -cha --max-depth=1 / | sort -h | grep -E "^[0-9\.]*[MG]" # say /var has taken up space du -cha --max-depth=1 /var | sort -h | grep -E "^[0-9\.]*[MG]"
- (no term)
- du options
- -c, –total
- have a grand total line at the bottom
- -a, –all
- write counts for all files, not just directories
- -m
- same as –block-size=1M
- -B, –block-size=SIZE
- SIZE can be K, M, G, T, P, E, Z, Y
- -s
- show total only
15.10.4.1. Large file deleted but still no space
- Delete file reserved by process
sudo lsof / | grep deleted- (no term)
- Either restart the service that is using the file or kill the process
- check inodes usage. Delete old files to release inodes
sudo df -i /
15.10.4.2. /tmp is mounted as overflow (often sized at 1MB)
It's likely due to /tmp was not specified as its own partition and the root filesystem filled up and /tmp was remounted as a fallback
After space has been cleared, just unmount the fallback and it should remount at tis original point.
sudo umount overflow # or sudo umount /tmp
15.10.4.3. Swap
free # if swap is not enabled, it shows Swap: 0 0 0 # or another to see if swap is on, is to list sudo swapon --show
- Swap partition is recommended
- Swap file
https://www.digitalocean.com/community/tutorials/how-to-configure-virtual-memory-swap-file-on-a-vps
linux:dd Swap file on Linux is a disk image e.g. /var/swap.img. Size is better 1 or 2x available system Ram. Run as root
cd /var touch swap.img chmod 600 swap.img # fill the file with zeroes for 1gb or 1024mb dd if=/dev/zero of=/var/swap.img bs=1024k count=1000 # or use this line to change file size sudo fallocate -l 1G /var/swap.img # initialize the swap filesystem mkswap /var/swap.img # enable swap file for the current boot swapon /var/swap.img # swapoff /var/swap.img # Make it ready at boot, add a line to /etc/fstab, be careful!! Don't overwrite, just add a new line echo "/var/swap.img none swap sw 0 0" >> /etc/fstab # in case you run sudo # echo "/var/swap.img none swap sw 0 0" | sudo tee -a /etc/fstab > /dev/null # set how likely Linux virtual memory manager to use swap. 100 means it uses swap as much as possible and leave memory free. # list all options sysctl -a # just VM options sysctl -a | grep vm. # or this way to see swappiness value cat /proc/sys/vm/swappiness # default seems to be 60 (you can also modify /etc/sysctl.conf file) sysctl -w vm.swappiness=30
/etc/fstab format
var/swap.img :: File system specifier :: For disk-based file systems, either a device file name (/dev/sda1), a file system label specification (LABEL=), or a devlabel-managed symbolic link (/dev/homedisk) none :: Mount point :: Except for swap partitions, this field specifies the mount point to be used when the file system is mounted (/boot) swap :: File system type :: The type of file system present on the specified device (note that auto may be specified to select automatic detection of the file system to be mounted, which is handy for removable media units such as diskette drives) sw :: Mount options :: A comma-separated list of options that can be used to control mount's behavior (noauto,owner,kudzu) 0 :: Dump frequency :: If the dump backup utility is used, the number in this field controls dump's handling of the specified file system 0 :: File system check order :: Controls the order in which the file system checker fsck checks the integrity of the file systems
15.10.5. mkdir
mkdir -m 777 my_dir_with_full_permissions
mkdir -pm 777 my_dir_with_full_permissions/{subfolder1,subfolder2}
15.10.6. File Permission
15.10.6.1. Return list of files in Octal
# full table of info stat filename # just the octal file permissions stat -c '%a' path/to/file for f in $(ls -a); do stat -c "%a %n" $f; done; # for a file or directory stat -c "%a %n" /var/www/html
15.10.6.2. Change permission
Octal mode
Permissions Binary Octal Description --- 000 0 No permissions –x 001 1 Execute only -w- 10 2 Write only -wx 11 3 Write and execute r-- 100 4 Read-only r-x 101 5 Read and execute rw- 110 6 Read and write rwx 111 7 full - Convert to Octal Mode
- Convert to Permissions Bit
The first digit of the 4 digits in Octal mode
Binary Octal Description 000 0 setuid, setgid, sticky bits are cleared 001 1 sticky bit is set 10 2 setgid bit is set 11 3 setgid and sticky bits are set 100 4 setuid bit is set 101 5 setuid and sticky bits are set 110 6 setuid and setgid bits are set 111 7 setuid, setgid, sticky bits are set - SUID or setuid
- If set, when the file will be executed by a user, the process will have the same rights as the owner of the file being executed
- If set, then replaces "x" in the owner permissions to "s", if owner has execute permissions, or to "S" otherwise. Examples:
-rws------both owner execute and SUID are set-r-S------SUID is set, but owner execute is not set
- Same as above, but inherits rights of the group of the owner of the file. For directories it also may mean that when a new file is created in the directory it will inherit the group of the directory (and not of the user who created the file)
- If set, then replaces "x" in the group permissions to "s", if group has execute permissions, or to "S" otherwise. Examples:
-rwxrws---both group execute and SGID are set-rwxr-S---SGID is set, but group execute is not set
- It was used to trigger process to "stick" in memory after it is finished, now this usage is obsolete. Currently its use is system dependant and it is mostly used to suppress deletion of the files that belong to other users in the folder where you have "write" access to
- If set, then replaces "x" in the others permissions to "t", if others have execute permissions, or to "T" otherwise. Examples:
-rwxrwxrwtboth others execute and sticky bit are set-rwxrwxr-Tsticky bit is set, but others execute is not set
- If set, then replaces "x" in the group permissions to "s", if group has execute permissions, or to "S" otherwise. Examples:
- owner-group-public
400- r---–— e.g. ssh private keys in ~/.ssh/ chmod:400
440- read for owner and group, but none for public chmod:440
555- read and execute (can't modify) for all chmod:555
700- rwx-–— full for owner chmod:700
755- rwxr-xr-x (e.g. wordpress directory) Default directory permission and default executable file permission
770- rwxrwx— chmod:770
775- rwxrwxr-x chmod:775
777- -rwxrwxrwx Directory's full permission chmod:777
600- -rw-–— only owner can read and write file chmod:600
644- -rw-r–r– (e.g. wordpress .php files)
660- rw-rw-— (e.g. wp-config.php) chmod:660
666- anyone can read and write. File full permission. chmod:666
664- rw-rw-r– chmod:664
- By default, umask is
0022- A file's full permission is 666. So new file has this default permission
- 666-022 = 644
- A directory's full permission is 777. So new directory has default permission
- 777-022 = 755
umaskumask 0022Alpha notation
# Remove write permission for all chmod a-w filename chmod ugo=rwx filename chmod u=rwx,g=rw,o=r filename chmod ug+w filename
- u
- user that owns the file
- g
- group that owns the file
- o
- other (everyone else)
- a
- all (everybody)
-R# In general, don't use -R but use find-chmod or find-xargs-chmod chain instead # - -R will # - Execute `fchmodat` system call for each file and sub folder. Hence, changing the ctime (file status modification time) for all its files and sub folders. # By using find-chmod or find-xargs-chmod, it finds the files and sub folders that need to be changed and then change the permissions cd /path/to/targetFolder # - `find -print0` # - `find` will separate each line of file path with a null character (a zero byte) # - `xargs` must then be called with `-0` or `--null` # - `xargs` # - Default command is `echo` # - `xargs -0` eq. `xargs -0 echo` # - `xargs echo` # - Result :: echo 'foundFile1.txt' 'foundFile2.txt' # - `xargs -I {} echo {}` # - `-I` :: Run the `echo` command for each record of the previous `find` command # - Result :: echo 'foundFile1.txt'; echo 'foundFile2.txt' # This is the best solution find . -type f ! -perm 666 -print0 | xargs -0 -I {} chmod 666 {} # chmod 666 foundFile1.txt # chmod 666 foundFile2.txt # ... # eq. # However, the list of files might be too long and error pops up # -bash: /bin/chmod: Argument list too long. find . -type f ! -perm 666 -print0 | xargs -0 chmod 666 # chmod 666 foundFile1.txt foundFile2.txt # eq. # Same caveat as the above find . -type f ! -perm 666 -exec chmod 666 {} + # eq. # Same caveat as the above chmod 666 $(find . -type f ! -perm 666) # eq. # `find .` can be `find /path/to/targetFolder`
- See wp:check file permissions
15.10.6.3. Change owner:group for a folder
# change a sub folder and all its files and directories sudo chown -R user:group aFolder # change current folder sudo chown -R user:group .
List files or directories that are not a user or group. See wp:file permissions
find . -type d -not -group test -o -type f -not -user test
15.10.6.4. Copy permissions and owners from one file to another
file2 will have the exact same owners and permissions as file1
chown --reference=file1 file2 chmod --reference=file1 file2
15.10.6.5. Create several users for SFTP and final file permissions are www-data linux:permission:users
If more than one users are needed, try linux:bindfs
Otherwise, add user to group www-data and change default wp:file permissions from chmod:644 files 2.5 directories to chmod:664 for files and chmod:775 for directories
This way new file will be the user not www-data but editing/uploading exising files will remain www-data:www-data
wp:check file permissions
15.10.7. Links
15.10.7.1. Example
# Hard links # reference a file in the filesystem # do not break if file is moved or deleted (hardlink will still hold the file) ln filetolink hardlink # hardlink has the same size as filetolink. Good to directly open or edit it cat hardlink # ls -al hardlink # -rw-r--r--@ # Symbolic link or Symlink or sym link, ~abc~ at user home folder, point it to existing folder ~xyz~ at user home folder. Reference a file path or a directory path # break if file is moved or deleted # ln -s filetolink symlink ln -s ~/xyz ~/abc # ls -al ~/xyz # lrwxr-xr-x # Relative Path :: you are at ~/abc/xyz, you want to create ~/abc/xyz/sym to point to existing folder ~/abc/efg/source ln -s ../eft/source/ sym # remove the symlink, not the files of it rm ~/abc/xyz/sym # Directly go to the actual folder by calling ~/abc cd -P ~/abc
- Move a directory to a different directory and make the original directory as a symlink to the new directory
# Check permission ll /path/to/source # Create the new directory with the same permission as the source mkdir -m 777 /path/to/destination # Make a copy of the source cp -a /path/to/source /path/to/source_backup mv /path/to/source/* /path/to/destination # remove the source directory rm -rf /path/to/source # create symlink ln -s /path/to/destination /path/to/source
15.10.7.2. Summary
- Can create symlink to non-existing file
- Options
- -s
- symbolic link instead of hard link
- -f, –force
- remove existing destination files
- On Windows
- See git:symlink
- To create a symlink on Windows, enable Developer Mode first and command prompt not PowerShell
mklink symlinkgoesto myfolder\mytargetfolderdel symlinkgoesto
- On Vagrant, see vagrant:symlink
15.10.8. Search files or directories, sort by size
- Return path of matched files or directories
-name-iname- Name case insensitive
use
-inameinstead of-name# trailing slash is required for the target folder path find ./targetFolder/ -type f -name "abc*"
? * []but not regex- used in
-name-iname-path
- used in
-type d -name "abc"find ./targetFolder -type d -name "abc*"ANDOR# AND # Start with abc and file extension is php find . -name 'abc*' -name '*.php' # Start with abc and file extension is not php find . -name 'abc*' ! -name '*.php' # OR find . -name '*.php' -o -name '*.txt' # -and -not can be used find . -name '*.php' -and -not -path *wp-includes*
K, M, G for kb, mb and gb
find . -type f -size +10000K
-mtime# Search text files that are modified less than 14 days ago find . -type f -mtime -14 ! -iname '*.jpg' ! -iname '*.png' # Modified between 50 and 100 days ago find . -type f -mtime +50 -mtime -100
-cmin# last 5 minute find . -cmin -5 # Last day 1440, Last 31 days 44640
-exec# {} is each result of find. # \; denotes the end of one command find . -size +100M -exec ls -lh {} \; # find and delete find . -type f -name "file-to-find" -exec rm -f {} \; # Multiple. The first one needs to return success in order for the next one to be run find . -name "*.txt" -exec echo {} \; -exec grep banana {} \; # Multiple. Add `-o` to make sure both commands are run find . -name "*.txt" \( -exec echo {} \; -o -exec true \; \) -exec grep banana {} \; # Plus sign find . -exec ls '{}' \; # executes: # ls file1 # ls file2 find . -exec ls '{}' \+ # executes # ls file1 file2
Get top 10 largest files in current folder
find . -type f -exec du -Sh {} + | sort -rh | head -n 10Depth Assume it's at home folder, and here's the structure
~/1/2/3 ~/a/1/2/3 ~/afile
At home folder,
find .will return./1 ./a /afile
The depth of
./1and./ais 1 While at home folder,find ~/will return/home/user/1 /home/user/a /home/user/afile
The depth of
/home/user/1and/home/user/arelated to~is 1 To find any folder that is named1but not at first levelfind ~ -mindepth 1 -iname 1
linux:wc
-not -path "a/path/*"# Exclude abc/.git directory find abc/ -type f -not -path "abc/.git/*" | wc -l # If abc is the current directory find . -type f -not -path "./.git/*" | wc -l # Count for files and folders find . | wc -l
-not -inamefind . -type f -not -name '*.jpg' -not -name '*.pdf' find wp-content/uploads -type f -not \( -name '*.jpg' -o -name '*.pdf' -o -name '*.gif' -o -name '*.png' \)
- wp:check file permissions
Sort by modified date newest first
find . -printf "%T@ %Tc %p\n" | sort -rn-iname --execfind -iname '*.jpg' -exec mv -t path/to/target/folder/ {} \+ find -iname '*.jpg' -exec cp {} path/to/target/folder/ \;
-perm# -perm mode :: exact match # rw- # rw- # r-- find . -perm 664 # -perm -mode :: match any of these # rw* # rw* # r** find . -perm -664 # -perm /mode :: Any of the permission bits mode are set for the file. # These are the same # Files which are writable by either their owner or their group. # The files don't have to be writable by both the owner and group to be matched; either will do. # 220 is # -w- # -w- # --- find . -perm /220 find . -perm /u+w,g+w find . -perm /u=w,g=w
15.10.9. Search Text in Files
- Regex is used. To match literal string, use
-F grep -Fr 'hello' .- (no term)
Basic Regular Expression
^ $ . [ ] - and * as Meta characters \d # any digit \D # anything not a digit \w # any word (alphanumeric + underscore) \W # anything not a word character \s # whitespace (space, tab, line break) \S # anything not whitespace
- Character Class
- anything inside a pair of brackets
[]- Special character classes
https://www.regular-expressions.info/posixbrackets.html#class
echo 'AaBbCcDdEe' | grep --color '[[:upper:]]'
[:alpha:][:digit:][:alnum:][:lower:][:upper:]- not including spaces. ASCII eq.
[!"\#$%&'()*+,\-./:;<=>?@\[\\\]^_‘{|}~] - space character (space, tab, new line)
- whitespace character
- printable characters, including space
- printable characters, not including space
- Control character (non-printing)
- Hexadecimal characters (0-9, A-F, a-f)
- Refer to js:regex:character class
- (no term)
Basic usage
# only files at this directory, not subdirectories grep 'abc' /root/dir # recursive (subdirectories as well) grep -r 'abc' /root/dir # for current directory grep -r 'abc' . grep --color -HRi "google\.com" /root/dir # Only file names grep --color -HRi "abc" /root/dir | cut -d: -f1 # Unique file names grep --color -HRi "abc" /root/dir | cut -d: -f1 | sort -u # Exclude file types grep --color -HRi --exclude=\*.{pdf,jpg,jpeg,JPG,mp3,png,bmp,sql} "abc" . # Exclude binary files grep --color -HIRin --exclude=\*.{pdf,jpg,jpeg,JPG,mp3,png,bmp,sql} "abc" . # Return 5 trailing lines and 10 leading lines for each match grep -A 5 -B 10 prefork.c /etc/httpd/conf/httpd/conf
- (no term)
Extended Regular Expression
(,),{,},?,+and|as extra Meta charactersgrep -E 'o' geeks.txt # 2 or more vowels grep -E '[aeiou]{2,}' geeks.txt # 1 to 2 vowels grep -E '[aeiou]{1,2}' geeks.txt # exactly 2 vowels grep -E 'el{2}' geeks.txt # Start of a word grep -E -i '\<h' geeks.txt # End of a a word grep -E 'y\>' geeks.txt # Word boundary :: match a whole word grep -E '\bGlenn\b' geeks.txt # Word boundary :: `way` must be part of a word or the whole word grep -E '\Bway\B' geeks.txt # OR ls -al | grep -i -E 'folder1$|folder2$'
- (no term)
- Options
- –color
- hightlight
-H, -h- return only the file names
-R, -r, --resursive- recursive
-w- whole word
grep -w apple fruit.txt
-v- inversion. Return lines that don't match
grep -v apple fruit.txt
-L- like
-v, suppress normal output, but print names of files that don't have a match -l- like
-Lbut print names of files that have a match -i- ignore case
-I- process binary files as if they don't contain matching data
-n- show line number
-c- show count only
-A <NUM>- number of trailing lines of text
-B <NUM>- number of leading lines of text
-F- literal text match instead of using regex by default
-E- extended regex
- (no term)
--exclude-dir=node_modules
- (no term)
GREP_OPTIONS- Set default options using linux:env
export GREP_OPTIONS='--color=auto --binary-files=without-match'then you don't have to provide-Iand--colorin command lines- Escape single quotes and backslash using backslash and separate options by space.
15.10.10. Translate
- Syntax
tr [OPTIONS]... SET1 [SET2]- (no term)
- Replace SET1 with SET2
echo '123456789' | tr '12345678' 'ABCDEFGH'- result
ABCDEFGH9 - Change to uppercase
echo 'hello world!' | tr [a-z] [A-Z]- eq. to
tr '[:upper:]' '[:lower:]'
- eq. to
- (no term)
Delete with
-decho 'abc123' | tr -d [:digit:] # abc echo 'abc123' | tr -dc [:digit:] # 123 # c option means NOT! inverse, contradict echo 'abc1233deee567f' | tr -s [:digit:] # abc123deeee567f # -s is squeeze consecutive matched characters echo 'abc1233deee567f' | tr -sc [:digit:] # abc1233de567 # c inverse again echo 'abc1233deee567f' | tr -ds [:digit:] [:alpha:] # abcdef # delete digits, squeeze on alpha characters echo 'abc1233deee567f' | tr -dsc [:digit:] [:digit:] # 123567 # c only applies to d here.. tr -d '\015\032' < windows_file > unix_file
15.10.11. column bash:column
Default delimiter is space
- Use colon
cat /etc/passwd | column -t -s ':'- Use tab
cat atabdelimited.csv | column -t -s $'\t'orcolumn -t -s '\t'Refer to linux:/etc/passwd
15.10.12. cut
- Extract columns from file
cut OPTION... [FILE]...- ">field 2 with delimiter tab
- Options
-d $'\t'- tab delimiter. It is also the default when
-fis used -f 2- field 2 after delimited
-c 2-10,72-- each line, grab character from 2nd to 10th position and 72 to the end. Index starts from 1
15.10.13. awk bash:awk
- https://www.tutorialspoint.com/awk/index.htm
- Space delimited by default
- Refer to bash:read
awk 'BEGIN {command1; command2;} /pattern/ {command3; command4;} END {command5; command6;}'BEGINrun AWK commands 1 and 2 before read- Read a line from input then execute AWK commands 3 and 4 using the line (
/pattern/to filter the line) - repeat
ENDrun AWK commands 5 and 6
awk 'BEGIN{printf "Col1\tCol2\tCol3\n"} {print}' marks.txt # use a command file awk -f command.awk marks.txt # command.awk content # BEGIN{printf "Col1\tCol2\tCol3\n"} {print} # define variables inside awk awk -v name=Jerry 'BEGIN{printf "Name = %s\n", name}' # pass variable to awk dockercontainer=lucee awk -v name="$dockercontainer" 'BEGIN {print name}' awk -v name="$dockercontainer" '{ if ( $1 == name ) print $2}' # print columns awk '{print $3 "\t" $4}' marks.txt # Using pattern to filter lines to run body commands awk '/somepattern/ {print $3 "\t" $4}' marks.txt # line count awk '/somepattern/ {print $3 "\t" $4; ++cnt } END {print "Count = ", cnt}' marks.txt # $0 is the line, length($0) awk 'length($0) > 18 {print length($0)}' marks.txt
15.10.13.1. Options
-F- change delimiter. Default is space
- tab
-F $'\t'or-F "\t"- colon or comma
-F ":"-F ','- (no term)
- Separator can be a single character or a regex
- (no term)
- Default space, multiple spaces are treated as one column
- Other single character
- e.g. for comma, multiple commas are treated as multiple columns
- (no term)
- To enable multiple commas as one comma/column, use
-F ',*'
15.10.13.2. AWK variables
- Print 17th column
cat access.log | awk {print $17}- (no term)
- Print all variables to
awkvars.outawk --dump-variables ''with no commands- Print one var
awk 'BEGIN {print "Arguments =", ARGC}' f1.txt f2.txt
- (no term)
- May change variables using
-v FS='\t'orBEGIN {FS= ","} - (no term)
ARGCandARGIND- multiple files as input
awk 'BEGIN {print "Arguments =", ARGC}' f1.txt f2.txtnumber of arguments 3 which areawk,f1.txtandf2.txt- only 1 argument
awk
ARGINDis the indexprint "Filename = ", ARGV[ARGIND]- multiline commands do not require
\
- multiple files as input
awk 'BEGIN { for (i = 0; i < ARGC - 1; ++i) { printf "ARGV[%d] = %s\n", i, ARGV[i] } }' f1.txt f2.txt
- FILENAME
- input field separator. Default space
- instead of using FS, a space-delimited list of field widths variable can be set to define fields
- output field separator. Default space
awk -F':' 'BEGIN{OFS="\t";} {print $3, $4}' /etc/passwdRefer to linux:/etc/passwd
- output record separator. Default
\n. This is used inprintin e.g. command 3 and 4. Manually add\nforprintf "%g/n",$1 - record separator. Default newline
\n - number of fields/columns in current line
- Print 3rd column to the last column
awk '{for(i=2; i<=NF; ++i) printf "%s ", $i; print ""}'awk '{for(i=3; i<=NF; ++i){printf("%s ", $i)}; printf("\n");}'
- Print 3rd column to the last column
- the number of the current line
- similar to NR but relative to the current file
- when it's set, pattern becomes case-insensitive
awk 'BEGIN{IGNORECASE = 1} /amit/' marks.txt
Variables that are functions related
- RSTART
- the first position in the string matched by match function
awk 'BEGIN { if (match("One Two Three", "Thre")) { print RSTART } }'
15.10.13.3. AWK operators
awk 'BEGIN { a = 50; b = 20; print "(a + b) = ", (a + b) }' # exponential awk 'BEGIN { a = 10; a = a**2; print "a =", a }' # or awk 'BEGIN { a = 10; a = a ^ 2; print "a =", a }' # result a and b are equal to 11 awk 'BEGIN { a = 10; b = ++a; printf "a = %d, b = %d\n", a, b }' # a = 11, b = 10 awk 'BEGIN { a = 10; b = a++; printf "a = %d, b = %d\n", a, b }' awk 'BEGIN { cnt = 10; cnt += 10; print "Counter =", cnt }' awk 'BEGIN { cnt = 10; cnt *= 10; print "Counter =", cnt }' awk 'BEGIN { cnt = 100; cnt /= 5; print "Counter =", cnt }' awk 'BEGIN { cnt = 100; cnt %= 8; print "Counter =", cnt }' # exponential awk 'BEGIN { cnt = 2; cnt **= 4; print "Counter =", cnt }' awk 'BEGIN { a = 10; b = 10; if (a == b) print "a == b" }' # != < <= >= # && || ! # ternary awk 'BEGIN { a = 10; b = 20; (a > b) ? max = a : max = b; print "Max =", max}' # unary # a = 10 awk 'BEGIN { a = -10; a = -a; print "a =", a }' # a = -10 awk 'BEGIN { a = -10; a = +a; print "a =", a }' # string concatenation awk 'BEGIN { str1 = "Hello, "; str2 = "World"; str3 = str1 str2; print str3 }' # regex # if line contains 9 awk '$0 ~ 9' marks.txt # if line does not contain 9 awk '$0 !~ 9' marks.txt
15.10.13.4. AWK functions
# arithmetic # atan2(y,x) # cos(expr) # sin(expr) # exp(expr) # int(expr) :: truncate to an integer value # log(expr) # rand() # srand([expr]) :: random number using seed value # sqrt(expr) awk 'BEGIN { PI = 3.14159265 x = -10 y = 10 result = atan2 (y,x) * 180 / PI; printf "The arc tangent for (x=%f, y=%f) is %f degrees\n", x, y, result }' # sort array based on values and change indexes to sequential integers starting with 1 awk 'BEGIN { arr[0] = "Three" arr[1] = "One" arr[2] = "Two" print "Array elements before sorting:" for (i in arr) { print arr[i] } asort(arr) print "Array elements after sorting:" for (i in arr) { print arr[i] } }' # sort array by indexes, after sort, the array value is the original index # gsub(regex, sub, string) # if string is omitted, $0 (current line) is used awk 'BEGIN { str = "Hello, World" print "String before replacement = " str gsub("World", "Jerry", str) print "String after replacement = " str }' # if string contains, the first occurance position is 1. Not contain is 0 awk 'BEGIN { str = "One Two Three" subs = "Two" ret = index(str, subs) printf "Substring \"%s\" found at %d location.\n", subs, ret }' # length(str) # match(str, regex) # returns the index of the first longest match of regex in string str. Return 0 if no match found awk 'BEGIN { str = "One Two Three" subs = "Two" ret = match(str, subs) printf "Substring \"%s\" found at %d location.\n", subs, ret }' # splits the string str into fields by regular expression regex and the fields are loaded into the array arr. If regex is omitted, then FS is used. awk 'BEGIN { str = "One,Two,Three,Four" split(str, arr, ",") print "Array contains following values" for (i in arr) { print arr[i] } }' # Decimal num = 123 # Octal num = 83 # Hexadecimal num = 291 awk 'BEGIN { print "Decimal num = " strtonum("123") print "Octal num = " strtonum("0123") print "Hexadecimal num = " strtonum("0x123") }' # one time substitution # sub(regex, sub, string) awk 'BEGIN { str = "Hello, World" print "String before replacement = " str sub("World", "Jerry", str) print "String after replacement = " str }' # substr(str, start, l) # returns the substring of string str, starting at index start of length l. If length is omitted, the suffix of str starting at index start is returned. # tolower(str) # toupper(str)
15.10.13.5. AWK pretty print
# horizontal tab awk 'BEGIN { printf "Sr No\tName\tSub\tMarks\n" }' # vertical tab (like IDE uses TAB to represent code hierarchy # Parent > child > grand child > grand grand child awk 'BEGIN { printf "Sr No\vName\vSub\vMarks\n" }' # backspace # Field Field Field Field 4 awk 'BEGIN { printf "Field 1\bField 2\bField 3\bField 4\n" }' # carriage return /r or ASCII value 13 or 0x0D awk 'BEGIN { printf "Field 1\rField 2\rField 3\rField 4\n" }' # form feed (page breaker) \f or ASCII value 12 or 0x0C awk 'BEGIN { printf "Sr No\fName\fSub\fMarks\n" }' # first character, if value is numeric, ASCII convert to a character. If string, output the 1st character # ASCII value 65 = character A awk 'BEGIN { printf "ASCII value 65 = character %c\n", 65 }' # %d and %i :: only print the integer part of a decimal number # Percentags = 80 awk 'BEGIN { printf "Percentags = %d\n", 80.66 }' # %e and %E :: [-]d.dddddde[+-]dd # Percentags = 8.066000e+01 # %E is # Percentags = 8.066000E+01 awk 'BEGIN { printf "Percentags = %E\n", 80.66 }' # %f :: [-]ddd.dddddd # Percentags = 80.660000 awk 'BEGIN { printf "Percentags = %f\n", 80.66 }' # %g :: use %e or %f, whichever is shorter # %G :: use %E or %f, whichever is shorter awk 'BEGIN { printf "Percentags = %g\n", 80.66 }' awk 'BEGIN { printf "Percentags = %g\n", "80.66%" }' # %o :: unsigned octal number # %u :: unsigned decimal number # %x :: unsigned hexadecimal number. %X uses uppercase # %% :: escape % and prints % # %10d :: 10 characters long # %05d :: with leading zeros and total length 5 characters long # %-5d :: When value length is less than 5, add spaces to the right so that the value is left adjusted. # %+d :: always add prefix to indicate whether number is positive or negative
15.10.13.6. AWK output redirection
# overwrite awk 'BEGIN { print "Hello, World !!!" > "/tmp/message.txt" }' # append awk 'BEGIN { print "Hello, World !!!" >> "/tmp/message.txt" }' # pipe awk 'BEGIN { print "hello, world !!!" | "tr [a-z] [A-Z]" }' # 2-way communication # don't understand yet...
15.10.13.7. AWK array
awk 'BEGIN { fruits["mango"] = "yellow"; fruits["orange"] = "orange"; fruits["pear"] = "pear"; delete fruits["pear"]; # multi dimensional (just use "...") md["0,0"] = 100; md["0,1"] = 200; print fruits["orange"] "\n" fruits["mango"]; }'
Array loop
awk 'BEGIN { arr[0] = 1; arr[1] = 2; arr[2] = 3; for (i in arr) printf "arr[%d] = %d\n", i, arr[i] }'
15.10.13.8. AWK if
awk 'BEGIN { a = 30; if (a==10) print "a = 10"; else if (a == 20) print "a = 20"; else if (a == 30) print "a = 30"; if (a==20) { print "1"; print "2"; } }'
15.10.13.9. AWK loops
# for loop awk 'BEGIN { for (i = 1; i <= 5; ++i) print i }' # while loop awk 'BEGIN {i = 1; while (i < 6) { print i; ++i } }' # do-while loop awk 'BEGIN {i = 1; do { print i; ++i } while (i < 6) }' # break awk 'BEGIN { sum = 0; for (i = 0; i < 20; ++i) { sum += i; if (sum > 50) break; else print "Sum =", sum } }' # continue awk 'BEGIN { for (i = 1; i <= 20; ++i) { if (i % 2 == 0) print i ; else continue } }' # exit awk 'BEGIN { sum = 0; for (i = 0; i < 20; ++i) { sum += i; if (sum > 50) exit(10); else print "Sum =", sum } }'
15.10.14. Folder structure
which tree # apt-get install tree # Mac: brew install tree # list only folders without files tree -d test/ # list everything tree test/ # List first level child folder only for the current folder tree -d -L 1 # List hidden files also # In no event, tree prints the file system constructs `.` (current directory) and `..` (previous directory). tree -a /path/to/dir
15.10.15. Compare files in 2 directories
diff --brief -Nr dir1/ dir2/ # -N shows difference of file existance diff -r dir1/ dir2/ # shows text comparison as well # compare filenames and directory names only diff <(cd dist && find . | sort) <(cd src && find . | sort) # if dist has test.txt but src doesn't, it will show # < ./test.txt
15.10.16. Compare text files
- Refer to brew:colordiff
diff file1.txt file2.txt # Header # 3d2 :: line 3 on the left file, line 2 on the right file # - d :: deletion # - a :: adding # - c :: changing # Body under Header # < means the line belongs to the left file # > means the line belongs to the right file diff -u file1.txt file2.txt # unified diff, which shows a bit differently # Use colordiff, it's better!
- Options
-i- case insensitive
-b- ignore chagnes to blank characters
-w- ignore all whitespace
-B- ignore blank lines
-r- recursive for directories
-s- show identical files
- Output options
-c- copied context
-u- unified context
-y- side by side
-q- only whether files differ
15.10.17. rm
- Options
-f,--force- silent
15.10.17.1. Delete using brace expansion
rm -rf abc.log.2012-03-{14,27,28}
# eq.
rm -rf abc.log.2012-03-14 abc.log.2012-03-27 abc.log.2012-03-28
15.10.17.2. Delete all files and sub directories
# delete the dir as a whole rm -rf /path/to/dir # delete all non-hidden files and sub directories in the dir but keep dir rm -rf /path/to/dir/* # delete files only and keep sub directories in the dir find /path/to/dir -type f -delete
15.10.17.3. Delete folders with a name
# Test it first find ~/www -iname WEB-INF -exec ls {} \; # Delete files in those folders find ~/www -iname WEB-INF -exec rm -rf {} \;
15.10.17.4. Delete all files and subdirectories of the current directory
find . -mindepth 1 -delete # -mindepth 1 is to leave the directory itself alone
15.10.18. ftp bash:ftp
15.10.18.1. Basics
ftp domain.com ftp 192.168.0.1 ftp user@ftpdomain.com # you get a prompt # ftp> # Commands ls cd afolder # set local directory where downloaded files will be stored on your local environment (not the ftp server) lcd /home/user/yourname # download a file get path/on/ftp/to/file # download multiple files mget *.xls # for download, better use bash:wget # upload a file that is in local folder put filename # upload a file that is in any folder on your local environment put /home/user/your/path/to/a/file # upload multiple files mput *.xls # close ftp connection bye exit quit # for help help
15.10.18.2. lftp
- http://lftp.yar.ru/lftp-man.html
sudo apt-get install lftp
To enable wildcard for rm, use glob
#!/bin/bash ftpsite="ftp://yourftp.com" ftpuser="b" # password doesn't seem to have to be escaped like Makefile ftppass="c" # make sure you are sure which folder to delete before run mdelete and rmdir # to delete ~/target/path/targetfolder lftp u $ftpuser,$ftppass $ftpsite -e 'set ssl:check-hostname false;set ftp:list-options -a;' <<EOMYF cd target/path ls -d */ .*/ ls -d targetfolder/ rm -r targetfolder rm -r targetfolder2 rm -r targetfolder3 glob -a rm -r ./*images* quit EOMYF exit 0
15.10.18.3. Options for ftp command
- -i
- turn off interactive prompting during multiple file transfers
- -n
- restrain from attempting auto-login upon initial connection
15.10.19. cp
Basics
# Copy a sub folder abc/ in the current directory to a new directory ~/xyz with everything preserved cp -a abc/ ~/xyz # Duplicate subfolder abc and name it xyz cp -a abc xyz # Copy multiple files to a directory cd /path/to/files cp -a 1.php 2.php /move/to/this/directory # Copy files and subdirectory of the source directory to the target directory cp -a /path/to/source/directory/. /path/to/target/directory/
- Options
-a-dR --preserve=all-d--no-dereference --preserve=links-P, --no-dereference- never follow symbolic links in SOURCE
-R,-r,--recursive- for directory
-i- default it will overwrite. Interactive to ask me
-f,--force- if an existing destination file cannot be opened, remove it and try again
- Copy the same file to multiple destinations use linux:tee
No need to create destination files
tee ~/dest1/a.txt ~/dest2/a.txt < ~/source/a.txt > /dev/null
To copy and exclude some directories use rsync
# At parent folder which is the parent for sourceFolder and destinationFolder rsync -Pavin sourceFolder/ destinationFolder --exclude node_modules # remove -n to actually run rsync
Copy and rename
ABC.*toDEF.*# test, return cp commands # for file in ABC.*; do echo cp -a "$file" "${file/ABC/DEF}";done for file in ABC.*; do cp -a "$file" "${file/ABC/DEF}";done
15.10.20. mv
# rename a file mv afile.txt afile2.txt # rename a directory mv path/to/dir path/to/nonexsitingdir # move a directory to another existing directory so that path/to/existingdir/dir mv path/to/dir path/to/existingdir # eq. mv path/to/dir path/to/existingdir/ # move a directory to the current directory so that path/to/destination_dir/dir cd path/to/destination_dir mv path/to/dir . # move a directory's files and subdirectories to another existing directory so that path/to/existingdir # path/to/dir remains mv path/to/dir path/to/existingdir # move a subdirectory's files and subdirectories to its parent directory mv -f subdirectory/{.,}* . # no overwriting mv -n path/to/dir path/to/existingdir # forece overwriting. This is the default!! mv -f path/to/dir path/to/existingdir # interactive asking for overwriting mv -i path/to/dir path/to/existingdir
15.10.21. mount
15.10.21.1. mount --bind
# Bind `source` directory to `destination` directory # `destination` directory content is set to the same as `source` directory # - Symlink is not created mount --bind path/to/source path/to/destination # See if a directory is a mountpoint # 1. path/to/destination is a mountpoint mountpoint path/to/destination # 2. `Mounted on` shows path/to/destination. # For non-mountpoint directory, `/` is shown df path/to/destination # Show the source directory the mountpoint points to findmnt path/to/destination # Unmount to remove the bind umount path/to/destination
15.10.22. tar
15.10.22.1. Compress and Read .tar file
# /home/targetFolder cd /home tar -cvzpf backup.tar.gz targetFolder # Read .tar file tar -tf backup.tar # Read .tar.gz file tar -tzf backup.tar.gz # Count number of files tar -tzf backup.tar.gz | wc -l # targetFolder/index.html # targetFolder/subfolder/index.html # ... # Exclude a folder tar -cvzpf backup.tar.gz targetFolder --exclude='wp-content/uploads' --exclude='anotherFolder' # Compress the current folder and save the archive in current folder cd /home/targetFolder tar -cvzpf backup.tar.gz . # Compress the current folder and save the archive in one level above tar -cvzpf ../backup.tar.gz . # The archive structure is # ./index.html # ./subfolder/index.html # ...
Options
- -z
- use gzip method
- -c
- create a new archive
- -C
- current directory
- -v
- verbose
- –list, -t
- List the contents
- -p
- preserve permissions (default for superuser)
- -f
- use archive file as result or input. It follows the output/input filename
- –exclude
- wrap in '' if it contains spaces
- –exclude='*'
- ignore 3 levels deep
Say you want compress the html folder at var/www/html, do this
tar -cvzpf mybk.tar.gz -C /var/www/html . # tar -tzf mybk.tar.gz # ./fileone ./filetwo # instead of tar -cvzpf mybk.tar.gz /var/www/html # tar -tzf mybk.tar.gz # /var/www/html/fileone /var/www/html/filetwo tar -cvzpf /path/to/mybk.tar.gz -C /var/www/html fileone # tar -tzf /path/to/mybk.tar.gz # ./fileone
Dry run :: use - for tar file name and pipe with wc
tar -cvzpf - /var/www/html | wc -c
15.10.22.2. Uncompress
# Read the tar file first tar -tf backup.tar.gz # If file structure is # targetFolder/index.html # targetFolder/subfolder.html ... # And put targetFolder so that /anotherfolder/targetFolder # Copy .tar.gz to /anotherfolder cd /anotherfolder tar -xvzf backup.tar.gz # extract only some files, only one `--wildcards` at a time tar -xvzf bk.tar.gz --wildcards '*.png' --ignore-case # If you don't want to copy .tar.gz to /anotherfolder, tar -xvzf /any/path/to/backup.tar.gz -C /path/to/anotherfolder/ # if you want to untar targetFolder only in the tar file tar -xvzf /any/path/to/backup.tar.gz targetFolder # if you want to untar targetFolder only and save the uncompressed as /anotherfolder/* not /anotherfolder/targetFolder tar -xvzf /any/path/to/backup.tar.gz --strip-components=1 # If file structure is # ./index.html # ./subfolder/index.html ... # Then copy backup.tar.gz to /anotherfolder/targetFolder cd /anotherfolder/targetFolder tar -xvzf backup.tar.gz # tar -xvzf /any/path/to/backup.tag.gz -C /anotherFolder/targetFolder # Uncompress file.gz, not file.tar.gz, gzip -d file.gz # The gz file will disappear
Options
- -x
- extract
- -C
- change the folder that the tar file will be uncompressed to. Without it, it will be the current folder
15.10.23. rsync
15.10.23.1. Basic
- Both client and server must have rsync installed
rsync -v
rsync [OPTION...] SRC... [DEST]- SRC
- If SRC ends with
/, it means only the files and subdirectories in that SRC directory. If it doesn't end with/and the SRC is a directory, then the whole directory is the SRC - DEST
- it doesn't matter to end the DEST directory with or without
/
rsync -rlvzuPn --size-only $SOURCE $DESTINATION-n- dry run
-z, --compress- compress data sent to destination machine
-u, --update- skip files that are newer in
$DESTINATION- See
--ignore-existing
- See
- Verbose and output
-v- verbose
--stats- show file-transfer stats: number of files, number of files transferred, Total file size in bytes, Total transferred file size
-i- Very useful! Display change-summary for all updates, also use with
--stats
-P--progress --partial- show progress and to resume interrupted transfers
-rlptgoD- recursive, symlink, same permissions, mod time, group/owner, device/special files
- recursive
- symlink
- preserve permissions. The receiving rsync sets the destinaion permissions to be the same as the source permissions
--no-p
- actually is very crucial. This tells rsync to transfer modification times along with the files and update them on the remote system. Unless
--size-onlyis used, missing-twill cause the next transfer to behave as if it used-I, causing all files to be updated- This command will not sync files with different permissions, ownership and modification time. Review the results and remove dry run
-nbefore running again - By default, rsync finds files that need to be transferred using a "quick check" algorithm that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated
- This modifies rsync’s "quick check" algorithm for finding files that need to be transferred, changing it from the default of transferring files with either a changed size or a changed last-modified time to just looking for files that have changed in size. This is useful when starting to use rsync after using another mirroring system which may not preserve timestamps exactly
- This command will not sync files with different permissions, ownership and modification time. Review the results and remove dry run
- Use it with
--size-only. Refer to pantheon:rsync-I,--ignore-times- Normally rsync will skip any files that are already the same size and have the same modification timestamp. This option turns off this “quick check” behavior, causing all files to be updated
- To set the group of the destination file to be the same as the source file. Only if dest is run as super-user
--no-g
- To set the owner of the destination file to be the same as the source file. Only if dest is run as super-user
--no-o
- To give new files the destination-default permissions (while leaving existing files unchanged), make sure
-pis off and use--chmod=ugo=rwXrsync -zar --no-p --no-g --no-o --chmod=ugo=rwX
- to use the default permissions of the destination dir
--no-g- to use the default group of the destination dir
--no-o- to use the default user of the destination dir
- hard set owner:group on dest.
rsync -rlvzPitog --chown=www-data:www-data /source /destination - is equivalent to
--devices --specials--devices- To transfer character and block device files to the remote system to recreate these devices
--specials- To transfer special files such as named sockets and fifos
- exclude a file or directory. Multiple
--excludecan be used--exclude='*.log'- by file extension
exclude multiple files and folders, use relative path without leading
./sources public_html/database.* downloads/test/*
- Only include log files
--include='*.log' --exclude='*' - skip updating files that already exist on the destination
--ignore-non-existing, --existing- skip creating files and directories that do not exist yet on the destination
- By default, rsync uses
~/.ssh/configon local to SSH into remote. Change the default e.g.-e 'ssh -p 2234'-e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'- See bash:sshpass
- On remote, specify which program (default
rsync). e.g. change it tosudo rsyncto elevate permissions- See 15.10.23.4
- Dangerous!
--delete- Delete extra files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronnized. By default, rsync does not delete files on the receiving side
15.10.23.2. Change summary, compare 2 directories
For comparing, you don't need -a specifically -pgoD so that left -rlt
You need -vzPin
Probably include -t and –size-only
rsync -rltvzPin wp-content/themes/my-theme/ sshconfighostname:/var/www/mylivesite/wp-content/themes/my-theme/
For current folder, use ./
Refer to bash:sshpass
.d..t...... ./ <f.st...... footer.php <f.st...... front-page.php .f..t...... functions.php <f.st...... header.php .f..t...... index.php .f..t...... page--coupon-image_tpl.php .f..t...... page--coupons_tpl.php .f..t...... style.css .d..t...... lb/ cd+++++++++ lb/api-localbusiness/ <f+++++++++ lb/api-localbusiness/cf-form-post.php <f+++++++++ lb/api-localbusiness/cf-form-post_rel0.php <f+++++++++ lb/api-localbusiness/cf-form-post_rel2.php .d..t...... lb/css/ .f..t...... lb/css/font.css <f.st...... lb/css/style.css .d..t...... lb/font/ .d..t...... lb/font/trade-gothic/ .f..t...... lb/font/trade-gothic/bold.eot .f..t...... lb/font/trade-gothic/bold.otf .d..t...... lb/img/ .f..t...... lb/img/alectra-logo.eps .d..t...... lb/img/bg/ .f..t...... lb/img/bg/lp-bg_001.jpg .f..t...... lb/img/bg/lp-bg_002.jpg .d..t...... lb/img/icons/ .f..t...... lb/img/icons/contact.png .f..t...... lb/img/icons/social-facebook.png .d..t...... lb/img/icons/coupon/ .f..t...... lb/img/icons/coupon/icon-01.svg .f..t...... lb/img/icons/coupon/icon-02.svg .d..t...... lb/img/logos/ .f..t...... lb/img/logos/brampton-hydro.png .f..t...... lb/img/logos/enersource.png .d..t...... lb/js/ .f..t...... lb/js/app.js .f..t...... lb/js/bicubic-interpolation.js
YXcstpoguax
Y is update type
- < means a file is being transferred to the remote (sent)
- > means a file is being transferred to the local (received)
- c means a local change/creation is occuring for the item (on the receiving side)
- h means the item is a hard link to another item
- . means the item is not being updated (though it might have attributes that are modified)
- * means the rest of the itemized-output area contains a message (e.g. "deleting")
X is file-type :: f for a file, d for directory, L for symlink, D for a device, S for a special file (named sockets and fifos)
cstpoguax are attributes.
- "." stands for no change.
- A newly created item replaces each letter with a "+"
- An identical item replaces the dots with spaces
- An unknown attribute replaces each letter with a "?" (could happen when talking to an older rsync)
- c: checksum is different
- s: file size is different and will be updated
- t: modification time is different and is being updated to the sender's value (requires -t or –times)
- T: modification time will be set to the transfer time
- p: permissions are different and are being updated to the sender's value (requires –perms)
- o: owner is different and is being updated to the sender's value (requires –owner)
- g: group is different and is being updated to the sender's value (requires –group)
- u: reserved for future use
- a: ACL information changed
- x: Extended attribute information changed.
15.10.23.3. Filter rules (INCLUDE/EXCLUDE Patterns)
If the pattern ends with a / then it will only match a directory, not a regular file, symlink or device
If the pattern starts with a / then it is similar to a leading ^ in regular expressions. lq/foorq would match at any point in the hierarchy.
oq*cq matches any path component but it stops at slashes
use ** to match anything, including slashes
oq?cq matches any character except a slash
oq[cq introduces a character class, e.g. [a-z] or
[[:alpha:]]
A trailing *** will match both the directory and everything in the directory e.g. foldername/***
15.10.23.4. Run rsync on remote using non-root SSH credentials
- Need to make sure the non-root SSH credentials can run
sudo rsync - Files uploaded to remote will have the default permissions, group and user of the destination remote dir
# --rsync-path 'sudo rsync' # --rsync-path="sudo rsync" # "--rsync-path=sudo rsync" # '--rsync-path=sudo rsync' rsync ... -zar --no-p --no-g --no-o --rsync-path='sudo rsync' ...
15.10.23.5. Examples
# change owner # - Dry, recursive, symlink, verbose, compress # - show progress and make it possible to re-rsync if connection is lost # - show changes in detail for each transferred file # - update time on destination # - don't transfer file if dest has mod time newer # - update owner and group on dest to be the same as source # - change owner:group on dest rsync -rlvzPituogn --chown=www-data:www-data wp-content/ remote:path/to/wp-content # Transfer everything from remote to local # - verbose # - show overall stats # - show individual file log # - show progress and re-sync possible # - compress # - files/directories are created on local using local file permission mask rsync -avziPn --stats $(devhost):$(devhostdir)/wp-content/uploads/ wp-content/uploads/ # Simple Download from Remote rsync -rlvzPitun --stats remote:path/to/site/sites/default/files/ sites/default/files rsync -a /vagrant/ma/test1 --files-from=/vagrant/ma/list.txt /vagrant/ma/test2 rsync -aI /vagrant/ma/test1 --files-from=/vagrant/ma/list.txt /vagrant/ma/test2 # list.txt # folder1/a.txt # abc.txt
15.11. Processes
ps -ef |
List of running processes |
sudo ls -l /proc/PID/exe |
file path for command, sub PID |
ps -f -p 123 |
List file of a running process |
ps aux |
List all processes for all users and |
| not started by a terminal (e.g. started by a daemon) | |
| and show user columns | |
ps options
- f or F
- full format. F for Ubuntu
- e
- all processes. same as -A
- p
- PID. same as p and –pid
- o
- specify format e.g. -o pid,ppid,pgid,sid
- ax
- list all processes from all users that are not started by a terminal (e.g. started by a daemon, background process that is removed from
jobs) - u
- Displays user-oriented output. This includes the USER, PID, %CPU, %MEM, SZ, RSS, TTY, STAT, STIME, TIME, and COMMAND fields.
- l
- Displays a long listing having the F, S, UID, PID, PPID, C, PRI, NI, ADDR, SZ, PSS, WCHAN, TTY, TIME, and CMD fields. Note :: -l doesn't include fields from -u
- w
- use with -f to show in unlimited width so that PPID and others are shown
15.11.1. pgrep, pkill, linux:kill
# Search a process by file name, return PID pgrep newcomgo # Kill a process by file name pkill newcomgo
15.11.2. Parent process linux:process:parent
Parent process id is PPID. PPID is 1 means parent is not found. Child process PID is 1234
# search in all processes ps -feww 1234 # just return ppid ps -o ppid= 1234 # return with custom format ps -eo pid,ppid,comm | grep 1234 # search in all processes that may start by daemon ps -axl | grep 1234
A package pstree can be installed and used
pstree -s -p 2072 outputs init(1)───pulseaudio(2061)───gconf-helper(2072)
15.11.3. Run executable in background for all terminal sessions
This should be enough. If not, use bash:screen :: nohup ./bin/newcomgo &
15.11.4. jobs
# Add ~&~ to put process list into background (tar -cf 1.tar /home/1; tar -cf 2.tar /home/2)& # run a script file in background ./test5.sh & # a job number and process number are returned # [1]+ 4867 # If terminal session is ended (SIGHUP), background processes will be stopped, too # Use nohup to block SIGHUP signals being sent to the process nohup ./test1.sh & # It produeces a file `nohup.out` in the directory that the script is in which contains STDOUT and STDERR. # No output to any terminals. # To prevent background processes from sending error to terminal ./test5.sh 2> /dev/null & # List processes that are running in background jobs # List PIDs jobs -l # Kill a process kill 1234 # or kill -KILL 1234 # above is eq. to below kill -9 1234 # Stop a process kill -STOP 1234 # Resume or continue a stopped process kill -CONT 1234 # job number can be used kill -STOP %2 kill -CONT %2 # remove or prevent a background process from being managed using `jobs` (using job number) and keep it running disown %2 # still can see the process ps -x | grep yourscriptname
15.11.4.1. Long running process
# Stop a long running process with C-z. A job number is returned # [1]+ Stopped bash message.sh # resume the stopped process in background so that new commands can be typed and executed but the process can still STDOUT bg %1 # bring the process to foreground fg %1 # another example # send process to background scp bigfile user:remote:/root & # bring the process to foreground so that you can login fg %1 # after login, pause the process using C-z # resume the paused process to background bg %1 # check status by bringing the process to foreground fg %1
15.11.4.2. Most recent, previously and multiple jobs
# multiple jobs fg %1 %2 bg %1 %2 # `jobs` shows + and - beside the most recent and previously bg jobs # target most recent `+` bg fg # target previously `-` bg - fg -
15.11.5. top
- First line
- current time, system up time, number of users logged in, load average (1, 5, 15 minutes; >2 is busy).
- Second line
- total processes: running, sleeping, stopped and zombie (have finished but parent process hasn't responded)
- Third line
- CPU utilization: us (user), sy (system), ni (nice: time running niced user processes), wa (time waiting for I/O completion), hi (time spent servicing hardware interrupts), si (software interrupts), st (stolen from this vm by the hypervisor)
- Memory usage
- Line 1 reflects physical memory: total, used, free and buffers
- Line 2 reflects virtual memory: total, used, free and cached
- Detailed for each process
- PID
- The process ID of the process
- USER
- The user name of the owner of the process
- PR
- The priority of the process
- (no term)
- NI: The nice value of the process
- (no term)
- VIRT: The total amount of virtual memory used by the process
- (no term)
- RES: The amount of physical memory the process is using
- (no term)
- SHR: The amount of memory the process is sharing with other processes
- (no term)
- S: The process status (D = interruptible sleep, R = running, S = sleeping, T = traced or stopped, or Z = zombie)
- (no term)
- %CPU: The share of CPU time that the process is using
- (no term)
- %MEM: The share of available physical memory the process is using
- (no term)
- TIME+: The total CPU time the process has used since starting
- (no term)
- COMMAND: The command line name of the process (program started)
- set sort, display more fields/columns
- Options
top -n 10 -o cpu -s 3 -U lili- top 10, order by cpu usage, update every 3 seconds, processes are of user lili
15.11.6. Multiple commands && vs || vs ;
Commands behind ; are always run Command behind && is run only when the previous returns success Command behind || is always run So || is the opposite of &&
false; echo 'yes' true; echo 'yes' false && echo 'yes' false || echo 'yes'
15.12. Memory
- Total memory
grep MemTotal /proc/meminfo- More info
cat /proc/meminfo- Memory usage
free -ltm -c 2 -s 2or justfree -mh- Delay 2 seconds and repeat 2 times
-c 2 -s 2- (no term)
- Options
- m
- in megabyte. e.g. -g
- t
- show Total
- l
- show Low and High
- h
- human readable
- Another way
vmstat -s- (no term)
- See memory usage for each process, use linux:top or
htop
Before in Ubuntu it shows 6 columns and the -/+ buffers/cache row total used free shared buffers cached Mem: 7916 7645 271 99 455 1764 -/+ buffers/cache: 5426 2490 Swap: 24999 805 24194
used = total - free
Now the columns are and no -/+ buffers/cache row total used free shared buff/cache available Mem: 3553 1192 857 16 1504 2277 Swap: 3689 0 3689
used = total - free - cached - buffers
Buffers are file system metadata and cache is pages with actual contents of files or block devices. Buff/cache will be freed up for newer applications that need memory to save I/O operations.
Linux OS always uses physical memory so free is always low.
A Linux system is really low on memory if the free value in -/+ buffers/cache: line gets low which is the available column in new Linux.
15.13. Service
15.13.1. SystemV init system :: other names SysVinit
- Characteristics
- synchronous
- things have to be executed in sequence
- List all services (on '+', off '-', managed by Upstart '?')
- Services defined as
/etc/init/*.confor/etc/init.d/*do not appear in this list
- Services defined as
serviceoperates on the daemon files in/etc/initor/etc/init.dwhich is an old init systemSet up an PHP script file as service
sudo nano /etc/init/myPHP.conf
start on startup stop on shutdown respawn script cd /path/to/the/script php myPHP.php end script
15.13.2. Upstart
- Characteristics
- Designed to replace SystemV
- event-based initialization program
- plug and play devices
- (no term)
- asynchronous
- (no term)
- automatic restart
- no update since 2014
- Designed to replace SystemV
15.13.3. SystemD :: System Management Daemon
- Characteristics
- parallel
List running services only with more detail
systemctl | grep 'php'systemctloperates on/etc/systemd. File in this folder will be used first otherwise fall back to old System V init- Some daemon files can do restart and it might be the preferred way to interact with service. e.g.
sudo /etc/init.d/jenkins restart Set up an PHP script file as a system Create
/etc/systemd/system/myPHP.servicefile or symlink to this directory https://maslosoft.com/blog/2019/07/10/running-php-script-as-a-system-service-in-ubuntu/[Unit] Description=Web Socket Server [Service] WorkingDirectory=/path/to/run ExecStart=/usr/bin/php comms_scheduled.php Restart=always [Install] WantedBy=multi-user.target
# Instruct SystemD that config was updated sudo systemctl daemon-reload sudo systemctl enable myPHP sudo systemctl start myPHP systemctl status myPHP # Looks like simply restarting the service can enable and start the new service systemctl restart myPHP
15.13.4. systemctl
sudo systemctl restart myapp.service sudo systemctl reload myapp.service sudo systemctl reload-or-restart myapp.service sudo systemctl enable myapp.service # /lib/systemd/system or /etc/systemd/system # /usr/lib/systemd/system # /etc/systemd/system/some_target.target.wants # enable and start. enable does not start the service in the current session
15.14. Network
All active NIC and IP addresses: ifconfig
All enabled and disabled NIC: ifconfig -a
View one NIC (interface): ifconfig eth0
Ethernet interface: eth0, eth1
Wireless network interface: wlan0, wlan1
Loopback interface for system internal communication: lo
Find default gateway ip route | grep default
15.14.1. /etc/network/interfaces file
For Ubuntu and Debian
iface eth0 inet static # only one address, call static # address 192.168.1.5 # netmask defines ip's within this range is considered under the same LAN # netmask 255.255.255.0 # When this machine talks to an ip out of this LAN, a gateway is needed # gateway 192.168.1.254 # DNS # dns-nameservers 192.168.1.254 8.8.8.8
Set interface to dhcp
auto eth0 iface eth0 inet dhcp
After changing interfaces file, turn off and on the interface
sudo ifdown eth0 && sudo ifup eth0
15.14.2. DNS
- Define DNS hosts
/etc/nsswitch.confand the line starting withhosts:defines which files Linux should look for the IP of a domainhosts: files mdns4_minimal [NOTFOUND=return] dns
First,
/etc/hostsfile127.0.0.1 localhost 127.0.1.1 your-computer-name # 127.0.1.1 is fine when the computer is a client # If the machine is a server and it has a static IP, should set it
Second, it's a list of DNS nameservers you define
- See a list of DNS nameservers
cat /etc/resolv.conf
- Not manually changable file
/etc/resolv.conf- (no term)
- A package called
resolvconfwhich handles the DNS nameservers. I guess if you change/etc/network/interfacesfile to usedns-nameservers, it will add nameservers to/etc/resolv.conf
Commands to check DNS are run based on
/etc/resolv.confgetent hosts google.com host google.com dig google.com
- dig can only follow through DNS management. If there's redirect done in Apache, Nginx or other codes, it won't follow through them
- Use 15.20.1 to track further
dig google.comordig google.com +shortdig google.com MXdig google.com ANYdig -x 72.30.38.140- DNS Propagation Checker
- Flush DNS
15.14.3. Flush DNS
- Ubuntu
sudo /etc/init.d/networking restart
15.14.4. Get Server's Public IP linux:get public ip
ip addr show eth0 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//' # or curl http://icanhazip.com
15.14.5. CIDR Netmask (Network Mask) CIDR netmask
- 10.0.75.5/32 = 10.0.75.5 AND 255.255.255.255 = 10.0.75.5 (a single host)
- 10.0.75.5/24 = 10.0.75.5 AND 255.255.255.0 = 10.0.75.0 (10.0.75.5 is in the 10.0.75.0 subnet)
- 10.0.75.5/16 = 10.0.75.5 AND 255.255.0.0 = 10.0.0.0
- 10.0.75.5/8 = 10.0.75.5 AND 255.0.0.0 = 10.0.0.0
Private IPv4 address spaces
- 10.0.0.0 - 10.255.255.254 = 10.0.0.0/8 (16 million number of addresses)
- 172.16.0.0 - 172.31.255.255 = 172.16.0.0/12 (255.240.0.0) (1 million number of addresses, 172.16.x.x - 172.31.x.x)
- 192.168.0.0 - 192.168.255.255 = 192.168.0.0/16 (65,536 number of addresses)
Internal network Class A :: 10.0.0.0 - 10.255.255.254 (10.0.0.0/8, total hosts: 16 million) Internal network Class B :: 172.16.0.1 - 172.16.255.254 (172.16.0.0/16, total hosts: 65,534) Internal network Class C :: 192.168.0.1 -192.168.0.254 (192.168.0.0/24, total hosts: 254)
15.14.6. Port
- If a port is in use
telnet localhost 9000, if it's connected, it's in use- It's better to use
telnet IP_address port_numberin an external computer to connect to a port to check if it's in use - To exit, first hit
C-]thenenter, intelnet>prompt, hitclose
- It's better to use
netstat- List all ports in use
netstat -lpornetstat -nlpto list in numeric form - Search a port
netstat -nlp | grep 8000 In Windows, to see if a port is being used in a local computer
netstat -an | find "8080" # From outside, just telnet host port (or telnet host:port on Unix systems) to see if the connection is refused, accepted, or timeouts
- netstat options
- l
- show only listening sockets
- p
- show PID and name of the program to which each socket belongs
- n
- show numerical addresses instead of symbolic host, port or user names
- t
- tcp
- u
- udp
- List all ports in use
lsof- List all command files (open files) that are using an Internet address (-i)
lsof -Pilsof -i :80
- Options
-Pis to print port number
[46][protocol][@hostname|hostaddr][:service|port]- 4 or 6 for IPv4 or IPv6
- List all command files (open files) that are using an Internet address (-i)
Kill processes that use the port
# this can also list PID's that use the port fuser 443/tcp -v # and kill fuser -k 443/tcp
If process keeps coming back, consider find the parent process and kill it
- Port range
- Use open ports that are in the range of 1024-49151
- 0-1023 are well known ports
- 1024-65535 are registered ports
- 49152-65535 are dynamic ports and should not be prescribed to a protocol (e.g. random)
- Use open ports that are in the range of 1024-49151
15.14.7. Wireless network adapters iwconfig
15.14.8. Gather monthly bandwidth
sudo apt-get install vnstat # check vnstat # tell it to monitor a network interface vnstat -u -i eth0 # it will create a database for it, check what interface is being monitored vnstat # start daemon sudo /etc/init.d/vnstat start # read stats for an interface vnstat -i eth0 # hourly, daily, monthly :: -h, -d, -m vnstat -h -i eth0 # live traffic :: -l # top 10 days with highest traffic :: -t
Info
- tx
- transmit
- rx
- receive
15.14.10. Netcat
# start listening and outputting data received on port 9006 nc -l localhost 9006 # keep listening # open another shell to make a curl request curl -F "textfield1=hello" -F "file1=@a.pdf" localhost:9006 # nc outputs the raw curl request # does not work on ports used by Apache # does not work for requests made from Postman..
15.14.11. Test email address if valid without sending
nslookup -q=mx gmail.com
You will see this:
Server: 192.168.1.200 Address: 192.168.1.200#53 Non-authoritative answer: gmail.com mail exchanger = 10 alt1.gmail-smtp-in.l.google.com. gmail.com mail exchanger = 20 alt2.gmail-smtp-in.l.google.com. gmail.com mail exchanger = 5 gmail-smtp-in.l.google.com. gmail.com mail exchanger = 30 alt3.gmail-smtp-in.l.google.com. gmail.com mail exchanger = 40 alt4.gmail-smtp-in.l.google.com. Authoritative answers can be found from: alt1.gmail-smtp-in.l.google.com internet address = 173.194.205.26 alt1.gmail-smtp-in.l.google.com has AAAA address 2607:f8b0:400d:c02::1a alt2.gmail-smtp-in.l.google.com internet address = 74.125.141.26 alt2.gmail-smtp-in.l.google.com has AAAA address 2607:f8b0:400c:c06::1a gmail-smtp-in.l.google.com internet address = 108.177.112.26 gmail-smtp-in.l.google.com has AAAA address 2607:f8b0:4001:c12::1b alt3.gmail-smtp-in.l.google.com internet address = 64.233.190.26 alt3.gmail-smtp-in.l.google.com has AAAA address 2800:3f0:4003:c01::1b alt4.gmail-smtp-in.l.google.com internet address = 209.85.203.26 alt4.gmail-smtp-in.l.google.com has AAAA address 2a00:1450:400b:c03::1a
Choose any one or the root one to telnet
telnet gmail-smtp-in.l.google.com 25 # if it's connected, it will say 220 # Then you first have to say HELO hi-whatever HELO hi # setup a From email, with <> brackets mail from: <youremail@gmail.com> # then specify the target email which you want to test rcpt to: <test@gmail.com> # If the target email is good, it will say 250 OK # 550 and its message may say Recipient address rejected: User unknown in virtual alias table quit
15.15. Firewall linux:firewall linux:ufw
- Should be installed by default
sudo aptitude install ufworsudo apt-get install ufw- Check if it's enabled
sudo ufw status verbose- Restart
sudo ufw disableandsudo ufw enableor just reloadsudo ufw reload- (no term)
sudo ufw reset- (no term)
- Refer to network:port to test
ufw comes with some profiles each of them is a setting for a service/app. Get all profiles sudo ufw app list
Available applications: Nginx Full Nginx HTTP Nginx HTTPS OpenSSH
Refer to nginx:ufw for further info.
Default setting is no restriction on outgoing connection, deny all incoming connection
Allow OpenSSH app/profile first sudo ufw allow OpenSSH then sudo ufw enable enable ufw
Review setting sudo vi /etc/default/ufw
Make sure IPV6=yes in setting.
Make sure DEFAULT_FORWARD_POLICY="ACCEPT" if docker is installed
Allow a service sudo ufw allow ssh
Allow a port sudo ufw allow 22/tcp
Allow FTP sudo ufw allow 21/tcp
Allow a port range sudo ufw allow 1000:2000/tcp
Allow an IP sudo ufw allow from 192.168.255.255
Allow HTTP web server sudo ufw allow 80/tcp
Allow SSL/TLS sudo ufw allow 443/tcp
Allow SMTP email sudo ufw allow 25/tcp
List all rules sudo ufw status numbered
Delete a rule sudo ufw delete [number]
Or you can see added rules in command style sudo ufw show added (even works when ufw is inactive)
Then you delete sudo ufw delete allow 22
15.16. Timezone, NTP
Get current timezone
date # show alphabetic timezone and numeric timezone date +"%Z %z"
sudo dpkg-reconfigure tzdata- Make sure Network Time Protocol daemon (NTP) is installed
which ntpd- Install
sudo apt-get updatesudo apt-get install ntp Do a sync
sudo service ntp stop sudo ntpd -gq sudo service ntp start
- Port 123 has to be open (but I think the default ufw setting is ok)
- Install
- For Debian/Ubuntu
cat /etc/timezone- file stores the timezone. Default is
Etc/UTC - (no term)
- All timezones are stored in
/usr/share/zoneinfo/$TZ- Toronto is
/usr/share/zoneinfo/America/Torontofile but it's a link to Montreal file
- Toronto is
- See Docker > timezone for changing timezone
15.17. Font linux:font
Install Source Code Pro
git clone --depth 1 --branch release https://github.com/adobe-fonts/source-code-pro.git /usr/local/share/fonts/adobe-fonts/source-code-pro/ sudo fc-cache -fv
- macos:font
15.18. Hardware
Most of devices: lshal
15.18.1. PCI: graphic card
# graphic card lspci -k | grep -EA3 'VGA|3D|Display' # more info about graphic card, get number [vendor:device] lspci -nn # e.g. # 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation G96 [GeForce 9500 GT] [10de:0640] (rev a1) sudo lspci -vvv -d 10de:0640
15.18.2. USB
- USB devices
lsusb- USB 3.0
- eq. to USB 3.1 Gen 1
- 5Gbps
- (no term)
- USB 3.1 Gen 2
- 10Gbps
- (no term)
- USB 3.2 Gen 1x1
- Same as USB 3.1 Gen 1 and USB 3.0
- (no term)
- USB 3.2 Gen 1x2
- 10Gbps
- (no term)
- USB 3.2 Gen 2x1
- Same as USB 3.1 Gen 2
- (no term)
- USB 3.2 Gen 2x2
- 20Gbps
- (no term)
- Thunderbolt is Intel and Apple only which only uses USB-C interface
- Thunderbolt 3
- Only 1 4K display
- A Thunderbolt 3 upstream/hub/dock can connect to 1 other Thunderbolt 3 downstream port
- Thunderbolt 4
- Compatible with USB 4
- Daisy Chain and other Thunderbolt special features
- 2 4K displays or 1 8K display
- A Thunderbolt 4 hub/dock can connect to 3 Thunderbolt downstream ports
- 40Gbps
- Thunderbolt 3
15.18.3. Monitor and Display
- Test refresh rate
- https://www.testufo.com/ghosting
15.19. Package, Repository
- A software repo, also called package source, is a network server, local directory or CD/DVD
- What software packages (versions, who package them) are available for download
- Debian
All repos
- Files
cat /etc/apt/sources.listand all files in/etc/apt/sources.list.d/- List all repos
grep ^ /etc/apt/sources.list /etc/apt/sources.list.d/*- (no term)
- Format
- Examples
deb http://site.example.com/debian distribution component1 component2 component3deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
deb-src http://site.example.com/debian distribution component1 component2 component3
- Archive type
deb- contains binary packages (pre-compiled)
deb-src- packages in source code, Debian control file (.dsc) and the diff.gz containing the changes needed for packaging the program
distributioncould be either- the release code name / alias e.g.
bionic - the release class e.g.
stable,testing
- the release code name / alias e.g.
- only include certain component(s)
Command to repo linux:add-apt-repository
sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -sc) \ stable"
- Repo URL
https://download.docker.com/linux/ubuntufile system structure- File
./dist/bionic/Releaseis copied to local when first time repo is registered
- File
./dist/bionic/InRelease sudo apt updatecompares localReleasefile with remoteInReleaseto check architectures, components, label, origin and suite (distribution)- To avoid checking,
sudo apt-get --allow-releaseinfo-change update. Not recommended
- Or remove the repo and add it back again. Refer to linux:ubuntu:ppa. Not recommended
- Files under
./dist/bionic/stable/actual files
- To avoid checking,
- File
Ubuntu
- Repos / Components
- Main
- free and open-source software supported by Ubuntu team
- Universe
- free and open-source software maintained by Ubuntu community
- Backports
- newest version of pkg
- Multiverse
- not free pkg (software restricted by copyright or legal issues)
- Restricted
- proprietary hardware drivers etc
- Canonical Partners
- software pkgs by Ubuntu for their partners
Sometimes
archive.ubunut.comdoesn't respond.. change it tous.archive.ubuntu.comsed -i 's/archive\.ubuntu\.com/us\.archive\.ubuntu\.com/' /etc/apt/sources.list # for EOL, change list sed -i 's/archive\.ubuntu\.com/old-releases\.ubuntu\.com/g' /etc/apt/sources.list sed -i 's/security\.ubuntu\.com/old-releases\.ubuntu\.com/g' /etc/apt/sources.list
15.19.1. Package Management
15.19.1.1. Debian only yum
yum check-update- only checks if any updates are available for installed packages only
yumdoesn't need toapt-get updatelike Ubuntu. In fact,yum updateupdates every currently installed package which is equivalent to Ubuntuapt-get update; apt-get upgrade- Try to avoid
yum upgradeas it forces the removal of obsolete packages. Same asyum update --obsoletes yum list treeto list package 'tree' info- If it returns error, it means the package tree is not installed
15.19.1.2. rpm Red-Hat Package Manager
- Low level
- RHEL (Red Hat Enterprise Linux) ,CentOS, Fedora
rpm -qa
15.19.1.3. dpkg lowest level Ubuntu, Debian
List installed packages: dpkg -l
How to read dpkg output
- First 3 columns show Desired action, Package status and Error flags
- Desired action
- unknown (u), install (i), hold (h), remove (r), purge (p)
- Package Status
- Not-installed
n- Config-files
c- Half-installed
H- Unpacked
U- Half-configured
F- Triggers-awaiting
W- Triggers-pending
t- Installed
i
- Error flags
- <empty> = (none), R = Reinst-required
If a package is installed: dpkg -l wget or dpkg -l | grep ^ii | grep -i wget
More info of an installed package :: dpkg -s openssh-client
See forward/reverse dependencies of a package use apt-cache showpkg javacc
List files owned by a package dpkg -L cron
Find which package owns a file dpkg -S /etc/crontab
15.19.1.4. apt-get apt-cache Ubuntu
- Since Ubuntu 16.04,
aptis introduced and it's a subset ofapt-getandapt-cache - Update list of available packages
/var/lib/apt/lists/apt updateapt-get update
It might return failed to fetch error. Might be due to linux:dns nameserver is changed to a local ip in a docker container in a docker network. Add a line
nameserver 8.8.8.8to/etc/resolv.conf. This way the/etc/resolv.confis not persisted. You can also config DNS in Docker daemonRUN echo "nameserver 10.1.1.1" >> /etc/resolv.conf && \ your-other-commands
apt-get install openssh-clientapt-get -s install openssh-clientapt-get upgradeapt-get -V -s upgradeapt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*- Official Debian and Ubuntu docker images automatically run
apt-get clean
- Official Debian and Ubuntu docker images automatically run
apt list --installed- For Ubuntu lower than 14.04, use
dpkg -lto list all installed packages on Ubuntu
- For Ubuntu lower than 14.04, use
- To see if a package is installed
apt -qq list xyzapt list -a --installed openssh-client
- Return info of a package before it's installed
apt show php- or
apt-cache show php - or
apt list -a --installed php(shows installed if it's installed)
apt-cache search phpapt-cache pkgnames php5apt-cache pkgnames- ~apt-cache showpkg php5-xmlrpc=
- Other packages must be installed before php5-xmlrpc (under Dependencies)
- Other packages depend on php5-xmlrpc (under Reverse Depends)
- Different versions of this package you can install (under Provides)
apt-cache policy openssh-client- https://packages.ubuntu.com/bionic/tmux
15.19.1.5. aptitude Highest Level Debian
It's not installed by default on Ubuntu It removes orphan packages automatically
List installed packages :: aptitude
See if a package is installed :: aptitude search package_name
- Before each package name, if you see `i` the package is installed
vcp- There might be a second character which indicates the action to be performed on the package.
15.19.2. Packages
15.19.2.1. openssh-client linux:package:openssh-client
Includes ssh, scp
apt-get update && apt-get -y install openssh-client
15.19.2.2. supervisord linux:package:supervisord
sudo apt-get install -y supervisor sudo service supervisor start # config is at /etc/supervisor/supervisord.conf # [include] # files = /etc/supervisor/conf.d/*.conf # after changing or adding config files supervisorctl reread supervisorctl update # all defined programs in config are run # go to CLI prompt, C-c to type exit to get out of the supervisorctl tool supervisorctl # available commands supervisor> help supervisor> stop nodehook supervisor> start nodehook
Built-in web interface
[inet_http_server] port = 9001 username = user # Basic auth username password = pass # Basic auth password
For example, create a webhooks.conf file to monitor a process
[program:nodehook] ;; nodehook is a name we give ;; the command to monitor command=/usr/bin/node /srv/http.js ;; change directory before running the above command directory=/srv ;; start the process when Supervisord starts (essentially on system boot) autostart=true ;; the program will be restarted if it exits unexpectedly autorestart=true startretries=3 stderr_logfile=/var/log/webhook/nodehook.err.log stdout_logfile=/var/log/webhook/nodehook.out.log user=www-data ;; environment variables to pass to the process environment=SECRET_PASSPHRASE='this is secret',SECRET_TWO='another secret'
15.19.2.3. sendmail
apt-get install sendmail # get the hostname of the machine/container hostname # say hostname is abc123 cat /etc/hosts # You should see something like this 127.0.0.1 localhost # add this line to the end 127.0.0.1 localhost localhost.localdomain abc123 # Run the sendmail config, answer Y to everything, sendmail will be restarted after sudo sendmailconfig # restart apache service apache2 restart # you may need to run sendmailconfig again to do the config # check other configs cat /etc/mail/sendmail.conf cat /etc/cron.d/sendmail cat /etc/mail/sendmail.mc # in case you want to restart sendmail service sendmail restart # or /etc/init.d/sendmail restart # run this to see which emails are sent/stuck in queue sendmail -bp
- Should setup logging in app level, such as php:ini:mail
/usr/sbin/sendmail
15.19.2.4. ssmtp
- ssmtp is a simple alternative to linux:sendmail and it doesn't depend on it
- sendmail can receive emails
- sendmail has mail queues
- ssmtp is good for only one user to send emails
15.19.2.5. postfix linux:postfix
- It's more sophisticated than sendmail.
apt-get install postfixwill remove sendmail automatically- https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-on-ubuntu-16-04
- https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-16-04
apt-get update && apt install mailutils # Default is Internet site. Use TAB and ENTER # System mail name: example.com // Use FQDN (bare domain) sudo nano /etc/postfix/main.cf
Setup to send only
Change to loopback interface, the virtual network interface that the server uses to communicate internally.
# inet_interfaces = all inet_interfaces = loopback-only
Similar to sendmail /etc/hosts setting
# mydestination = $myhostname, example.com, localhost.com, , localhost mydestination = $myhostname, localhost.$mydomain, $mydomain
Restart postfix
/etc/init.d/postfix status postfix status postfix start sudo systemctl restart postfix # Test echo "This is the body of the email" | mail -s "This is the subject line" myemail@a.com # From: sammy@example.com where sammy is Linux username and example.com is the server's hostname
Forward emails sent to root
sudo nano /etc/aliases # With the following, system generated emails are sent to the root user. postmaster: root # Add root: myemail@a.com # restart sudo newaliases # Test echo "This is the body of the email" | mail -s "This is the subject line" root
15.19.2.6. rtorrent
apt-get install rtorrent
Press Enter to enter and you will see load.normal> at the bottom
Paste the link and press enter to load the torrent file, to download it, press up or down arrow and C-s to start the download.
C-q :: quit. Execute twice, it shuts down without sending a stop signal C-s :: start download C-d :: Stop an active download or remove an already stopped download C-k :: stop and close an active download C-r :: hash check a torrent before upload/download begins Left or right arrow :: redirect to previous or next screen Up or down arrow :: to select a torrent file. Selected one will have * in front.
15.19.2.7. Tmux bash:tmux
sudo apt install tmux # version (2.6) tmux -V # start tmux to create a new session tmux
- 0 is session number, 1 is window number
[0] 1:yourname@your-machine-name:~*- review keyboard shortcuts, press q to exit
C-b ?- (no term)
- default bind-key is
C-b
Window
- Create a new window
C-b c- If you create 3 windows, the bottom looks like
[0] 0:bash 1:bash- 2:bash*- Rename current window
C-b ,- Switch to another window
C-b WindowNumber- Switch to next window
C-b n- Switch to prev window
C-b p- Switch to last window
C-b l- Close current window
C-b &- Show a list of all windows
C-b w- (no term)
*shows which window is active and-shows previous active
Pane
- Split current window horizontally into panes
C-b "- Split current window vertically into panes
C-b %- Switch to another pane
C-b oorC-b ArrowKeyorC-b ;- Move content in another pane to the current pane
C-b C-o- Resize current pane by increments
C-b C-ArrowKey- Promote a pane to a new window
C-b !- Put a clock into current pane
C-b tand press any key to remove the clock- Close current pane
C-b x
Session
- Detach the tmux session and go back to original shell
C-b d- List tmux session
tmux ls- Reattach a tmux session
tmux attachortmux attach 1- Set tmux session name
C-b $- Swtich to another tmux session
C-b s- Close a session
- either close all windows of that session or
tmux kill-session -t0
Scroll Mode
- Enable navigation mode
C-b [, use arrow keys andqto quit
Copy Mode
- Enable mode
C-b PgUp
Command line
- Enter command line
C-b :- Quit command line mode
- Set option globally
:set-option -g monitor-activity on
Customize ~/.tmux.conf
# don't allow tmux to automatically rename my already named windows set-option -g allow-rename off # Change status bar color # set -g status-bg cyan # set bind-key to C-a set-option -g prefix C-a unbind C-b
15.19.2.8. OBS Studio - Open Broadcaster Software linux:app:obs-studio
- Install
Ubuntu 18.04
# Install :: https://github.com/obsproject/obs-studio/wiki/Install-Instructions#linux # requires xserver-xorg. xserver-xorg-core must be 1.18.4+ apt list -a xserver-xorg apt list -a xserver-xorg-core # requires ffmpeg apt list -a --installed ffmepeg sudo apt install ffmpeg # install obs-studio sudo add-apt-repository ppa:obsproject/obs-studio sudo apt-get update sudo apt-get install obs-studio
- Usage
https://www.becomeablogger.com/obs/
- Scene
- a scene contains sources
- (no term)
- Source
- WebCam
- Window
- Right click > Preview scaling
- Scale to Window
- Right click > Transform
- Fit to screen
- Image
- you can use a text file and modify the file while recording
- (no term)
- Quickly switch between scenes
- Scene1
- Webcam Only
- Scene2
- Webcam + Image
- Scene3
- Browser + Webcam in small window
- Studio mode
- adjust one scene before it goes live
- click on Scene1 and go to Studio Mode. Now the recording is showing Scene1
- In Studio Mode, create Scene3 and adjust the browser. Scene3 is not in the recording
- Then click on Transition insdie Studio Mode to switch to Scene3. Scene3 is live and Scene1 becomes preview
Settings
- General
- Show confirmation dialog when starting streams
- Show confirmation dialog when stopping streams
- Output
- Advanced > Recording
- Change save directory
- Audio
- Disable Desktop Audio Device and Mic/Auxiliary Audio Device
- Video
- Base (Canvas) Resolution
- your monitor resolution
- Output (Scaled) Resolution
- your monitor resolution to have perfect quality
- Config
File > Settings
- Video
- Change Base (Canvas) Resolution and Output (Scaled) Resolution to 1920x1080 even though my screen is 2560x1440 (16:9)
- Lanczos
- 30 (changed from 60)
- Output
- Recording
- Recording Format
- flv
- Encoder
- x264
- Rescale Output
- 1920x1080
- Custom Muxer Settings
- blank
- Rate Control
- CRF (changed from CBF) for CPU Usage Present ultrafast
- CRF
- 15 (should be between 15 and 25)
- Profile
- none
- Tune
- none
- x264 Options
- blank
- Recording
- Video
- Audio Filters
- Compressor
- Noise Suppression
15.19.3. Ubuntu troubleshooting
15.19.3.1. Unmet dependencies
Run these
sudo apt-get clean sudo apt-get update sudo apt-get -f install # check if it returns error that about disk full error # fix disk full error first # install the package again sudo apt-get install make
If it still doesn't work or it has error, run these
sudo dpkg --configure -a sudo apt-get -f install
If the output is below, it means it failed.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Next solution is to run
sudo apt-get -u dist-upgrade
If it shows any held packages, it is best to eliminate them. Packages are held because of dependency conflicts that apt cannot resolve. Try this command to find and repair the conflicts:
sudo apt-get -o Debug::pkgProblemResolver=yes dist-upgrade
If it cannot fix the conflicts, it will exit with: 0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
Delete the held packages one by one, running dist-upgrade each time, until there are no more held packages. Then reinstall any needed packages. Be sure to use the –dry-run option, so that you are fully informed of consequences:
sudo apt-get remove --dry-run package-name
Since removing the package you are trying to install may not be ideal, you might also try finding a repository that has the packages you need to satisfy the dependencies.
Finally, if all else fails, you can attempt to satisfy the dependencies yourself, either by finding and installing the necessary packages, or by installing them from source and then creating “deb” packages for them.
https://askubuntu.com/questions/140246/how-do-i-resolve-unmet-dependencies-after-adding-a-ppa
15.19.3.2. /boot directory is full linux:disk full
If it's full 100% df -h, all apt commands will not run.
Run this to get installed kernels except the currently running one.
sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r` # or simply sudo dpkg --list 'linux-image*' for all kernels
Also keep track of the 2 newest versions. uname -r to check the current kernel version. e.g.
linux-image-4.4.0-31-generic linux-image-4.4.0-36-generic linux-image-4.4.0-38-generic linux-image-4.4.0-42-generic linux-image-4.4.0-45-generic linux-image-4.4.0-47-generic linux-image-4.4.0-51-generic linux-image-4.4.0-53-generic linux-image-extra-4.4.0-31-generic linux-image-extra-4.4.0-36-generic linux-image-extra-4.4.0-38-generic linux-image-extra-4.4.0-42-generic linux-image-extra-4.4.0-45-generic linux-image-extra-4.4.0-47-generic linux-image-extra-4.4.0-51-generic linux-image-extra-4.4.0-53-generic
Current kernel version is 4.4.0-34-generic
Remove other kernels except the current one and the 2 newest ones.
Before delete, check which will be deleted first
ll /boot/*-4.4.0-{31,36,38,42,45,47}-*
sudo rm -rf /boot/*-4.4.0-{31,36,38,42,45,47}-*
df -h will show the space is reduced. Fix apt-get
sudo apt-get -f install
If you run into an error that includes a line like "Internal Error: Could not find image (/boot/vmlinuz-3.2.0-56-generic)", then run the command sudo apt-get purge linux-image-3.2.0-56-generic (with your appropriate version).
Now apt-get should work. Finally, sudo apt-get autoremove to clear out the old kernel image packages that have been orphaned by the manual boot clean.
Run sudo apt-get update and maybe sudo apt-get upgrade after.
15.20. curl, wget
15.20.1. curl
15.20.1.1. Basic
apt update && apt install -y curl
#!/bin/bash dlDir=/home/lili/download/ dlFileName= ${dlDir}123.xml curl "http://abc.com/123.xml" -L -o ${dlFileName}
15.20.1.2. Return HTTP response header only
curl -m 120 -IL "http://url.com" >> /path/to/log.log
e.g Drupal Cron URL response status
#!/bin/bash curl -sL -o /dev/null -w "%{http_code}\n" "http://drupalcronurl" >> /path/to/your/cronlog
15.20.1.3. Download with HTTPS or HTTP authentication
- Popup form or server authentication
curl https://a.com/b.aspx -u user:password -L -o /path/to/xml.xml
15.20.1.4. cURL download only after or before a date
Works for ftp and Http
fileLocalTime=$(date -r "$localfile") curl $remote_file_path -u $user:$password -o $localfile -R -z "${fileLocalTime}" # default is "after" # add a dash before date string will result in "before". fileLocalTimeNew=$(date -r "$localfile") if [ "$fileLocalTime" == "$fileLocalTimeNew" ]; then echo "the same" else echo $fileLocalTime echo $fileLocalTimeNew fi
15.20.1.5. HTTP POST
- Refer to bash:pass arguments
# default POST is application/x-www-form-urlencoded curl -X POST -d "test=that&test2=that2" http://.. # explict -H 'Content-Type: application/x-www-form-urlencoded' # with a data file curl -X POST -d "@data.txt" http://.. # json # keep value to be a single line # I haven't figured out how to escape newline for json yet.. to='a@a.ca' from='b@a.ca' json_fmt='{"personalizations": [{"to": [{"email": "%s"}]}],"from": {"email": "%s"},"subject": "%s","content": [{"type": "text/plain", "value": "%s"}]}' data_json=$(printf "$json_fmt" "$to" "$from" "$subject" "$content") curl -X POST -H 'Content-Type: application/json' -d "{\"test\": \"that\"}" http://.. curl -X POST -H 'Content-Type: application/json' -d '{"test":"that"}' http://.. curl -X POST -H 'Content-Type: application/json' -d "$data_json" http://.. # with a data file curl -X POST -d "@data.json" http://..
15.20.1.6. Use different IP address from DNS
a.ca by DNS points to 1.2.3.4 IP, change it to a different IP 2.2.3.4 temporarily.
curl -L -v -I --resolve 'a.ca:80:2.2.3.4' http://a.ca # https://a.ca still points to 1.2.3.4 as port 80 does not match the request a.ca@443 curl -L -v -I --resolve 'a.ca:80:2.2.3.4' https://a.ca # Use this instead curl -L -v -I --resolve 'a.ca:443:2.2.3.4' https://a.ca # sub domain curl -L -v -I --resolve 'www.a.ca:443:2.2.3.4' https://www.a.ca # --resolve <host:port:address> force resolve of HOST:PORT to ADDRESS
15.20.1.7. Fetch web pages and save as HTML files
Uses bash:array
#!/bin/bash set -o history -o histexpand declare -A fileMaps=( [https://a.com/]='pages/index.html' \ [https://a.com/page1]='pages/page1.html' \ [https://a.com/page-2]='pages/page-2.html' \ ) declare -a curlArgs=('-L' '-H' "Connection: keep-alive" '-H' "Pragma: no-cache" '-H' "Cache-Control: no-cache" '-H' "Upgrade-Insecure-Requests: 1" '-H' "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" '-H' "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" '-H' "Accept-Encoding: gzip, deflate, br" '-H' "Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh-TW;q=0.7,zh;q=0.6") for K in "${!fileMaps[@]}"; do curlArgsTemp=("$K" "${curlArgs[@]}") curlArgsTemp=( "${curlArgsTemp[@]}" "-o" "${fileMaps[$K]}" ) curl "${curlArgsTemp[@]}" done # echo !!
15.20.1.8. Fetch as Googlebot curl:ua:googlebot
Googlebot's user agent: https://support.google.com/webmasters/answer/1061943
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -v https://a.ca
15.20.1.9. Options
| -L | Follow through all redirects |
| -o | redirect STDOUT to a file |
| -O | save file using same file name in current folder |
| -I | get header response only |
| -m | timeout in seconds |
| -H | request header. -H "Accept-Charset: ISO-8859-1,utf-8;q=0.7" |
| –limit-rate | limit download speed: –limit-rate=200k. use k, m or g |
| -s | suppress STDOUT. -svo /dev/null show req/resp headers only |
| -w | rewrite STDOUT |
| -R | Keep remote file timestamp. –remote-time |
| -z | cURL download only after or before a date |
| -d | send POST data |
| -A | --user-agent |
| -v | verbose |
| -b, –cookie | -b "name1=v1; name2=v2" |
15.20.2. wget bash:wget
Compared to curl, wget's advantages are:
- recursive
- continue downloading after broken transfer
- Specify cookies in get request
- Has redirect-following
- Capture response such as time stamping from the remote resource.
Disadvantages are only HTTP/HTTPS/FTP, Basic auth over HTTP proxy, no SOCKS support
wget [option]… [URL]…
Run a wp cron:
# /dev/null wget -O /dev/stdout -o /dev/stdout "https://mysite.ca/wp-cron.php?import_key=123&import_id=8&action=processing"
http://[username:password@]host[:port]/directory/file ftp://[username:password@]host[:port]/directory/file
Download all subfolders and files of an FTP folder
cd ~ wget --user=hello --password='mypassword' -cr ftp://server/path/to/test # Default depth is 5, change it wget -r -l 2 ftp://... # for infinite depth :: -l inf # path/to/test is 3 depths, by default, 2 more levels can be downloaded # ~/server/path/to/test/* will be downloaded # -nH ~/path/to/test will be downloaded # -nH --cut-dirs=1 ~/to/test will be downloaded # -nH --cut-dirs=2 ~/test will be downloaded # --cut-dirs=1 ~/server/to/test will be downloaded # --ftp-user or --http-user can also be used. There's no difference # Ignore folders (not file) wget -cr -X path/to/folder/ignore/ ftp://server/path/to/folder/ # folder path/to/folder/ignore/ will not be downloaded # wildcards can also be used wget -cr -X path/to/folder/ignore*/ ftp://server/path/to/folder/ # ignore multiple folders -X path/to/folder/ignore1,path/to/folder/ignore2 # Only include certain folders -I, exactly the same as -X # Only download files -A -A books*,zelazny*196[0-9],gif,jpg gif and jpg are normal letters, they match the ending part of a file (not exactly file extension) # Ignore certain files -R, exactly the same as -A # -nH :: --no-host-directories :: use it with --cut-dirs. e.g. download http://a.com/ by default creates a.com/ directory # -P :: --directory-prefix :: Default is . (the current directory) # -N to retrieve timestamps so if local files are newer, skip download. # -c :: enable continue download # --limit-rate=20k :: limit download speed to 20kb/s, m or g can also be used # -b :: put command run in background # -O path/to/local/file :: output to a local file. Change downloaded filename # Logging # -v :: default # -q :: complete quiet # -nv :: only error and basic info are printed
15.21. Bash File, Input and Output
15.21.1. Version echo $BASH_VERSION
15.21.2. Shebang
#!/usr/bin/bash- Absolute path to a shell
#!/usr/bin/env php- Environment variables are loaded and use the default shell to run command
php- Environment variable
SHELLdefines the default shell and it's loaded in the files listed in environment variablePATH
- Environment variable
- Environment variables are loaded and use the default shell to run command
15.21.3. Last directory ~-
cd ~/abc cd xyz echo ~-
15.21.4. Directory of the curren script file
# Normal usage when there is no symlink # Result has no trailing `/` DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" # When symlinks are involved SOURCE="${BASH_SOURCE[0]}" while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null 2>&1 && pwd )" SOURCE="$(readlink "$SOURCE")" [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located done DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null 2>&1 && pwd )"
15.21.5. range bash:range
echo {1..100} # 100 numbers from 1 to 100 touch file_{1..100} # add padding (one zero in front is enough) echo {01..100} # 01 02 ... 100 # range with increment other than 1 echo {1..10..2} # 1 3 5 7 9 echo {1..10..3} # 1 4 7 10 echo {A..Z} # A to Z echo {A..z} # A to z echo {a..Z} # not a to Z! # a ` _ ^ ] [ Z echo {w..d..2} # w u s q o m k i g e # customize range touch {apple,banna,cherry,durian}_{01..100}{w..d}
15.21.6. File Descriptors, Redirection Operator, Piping
- Use linux:tee to redirect output to a file and display output at the same time
- File descriptors
- First 3 are reserved
STDINStandard inputSTDOUTStandard output e.g. echoSTDERRStandard errorUp to 9 open file descriptors can be used at a time
# Open a custom file descriptor to a file exec 3>customLog # redirect output to file descriptor 3. # `&` on the right is just required for syntax # `&` on the left means both 1 and 2 file descriptors echo "This goes to file descriptor 3" >&3 # Open a custom file descriptor and redirects to STDOUT exec 4>&1 # If you want to change the reserved file descriptors. Always backup and change back exec 5>&1 exec 6>&2 # Backup STDIN is a little different exec 7<&0
>and<# `&` on the right is just required for syntax # `&` on the left means both 1 and 2 file descriptors # Redirect stdout, creates if not exist or overwrites a file ls -al goodfile > stdoutlogfile # same as 2>, 3> etc. # Redirect stdout, creates if not exist and appends to a file ls -al goodfile >> stdoutlogfile # same as &>>, 2>>, 3>> etc. # Redirect stderr only ls -al badfile 2> errorlogfile # Redirect stdout and stderr to different files ls -al goodfile badfile 2> errorlogfile 1> normalOutput # Redirect stdout and stderr to the same file ls -al goodfile badfile &> errorAndOutput # Redirect stdout and stderr to the same file by appending ls -al goodfile badfile &>> errorAndOutput # suppress stdout and stderr ls -al goodfile badfile &> /dev/null # Usage of >, &>, 2>>, etc # On the left, it's an output not a file # Usage of < # On the right, it's a file not an output sort < fruit.txt wc < fruit.txt # clear a file's content, perfect when sudo is not needed < clearThisFile.txt
-
# on the left, output # on the right, command that takes input echo "Hello" | wc
15.21.6.1. tee
Basics
# Redirect stdout of a commannd to a file and display stdout date | tee testfile # Redirect stdout and sterr of a command to a file and display as stdout date 2>&1 | tee testfile # tee can be used to bypass pagination (press q to quit) man ls | tee # Redirect stdout to a file and pass stdout to another command as stdin date | tee testfile | less # Capture stdout of a command to a variable output=$(date) # Capture stdout and stderr of a command to a variable output=$(command_which_has_stdout_and_stderr 2>&1) # Redirect stdout of a command to one or multiple files and save stdout to a variable # a file must be behind a tee command # By default tee overwrites the file. Use -a to append to the end $output = $(date | tee -a testfile1 testfile2) # Redirect stdout to a variable and display to terminal in real time (not to stdout!) # Works on Ubuntu and RedHat $output = $(date | tee /dev/tty) # Redirect one file descriptor (M, default 1) to another file descriptor (N) M>&N # Redirect a manually made error echo "This is an error" >&2 # Change a file descriptor to a file exec 2>errorLogFile echo "This is STDOUT and STDOUT is not changed to a file yet" exec 1>stdOutFile echo "This doesn't display as STDOUT is redirected to a file" echo "This is an error and in errorLogFile" >&2
- linux:cp:multiple destinations
15.21.7. declare bash:declare
# integer declare -i d=123 # read only, var can't be changed declare -r e=456 o=hello # uppercase -u, lowercase -l declare -u upper=$o
15.21.8. Array bash:array
# define empty array myarray=() # define indexed array with values myarray=( one two three four five ) declare -a curlArgs=('-H' "keyheader: value" '-H' "2ndkeyheader: 2ndvalue") # run `help declare` # declare -a for indexed array # declare -A for associative array # define array with strings myarray=( 'Hello World' 'Another item' ) # Get at index 0 echo ${myarray[0]} # print array as a whole delimited by space echo ${myarray[@]} # print array as a whole delimited by newline printf "%s\n" "${myarray[@]}" # Convert an array to string variable delimited by newline printf -v stringvar "%s\n" "${myarray[@]}" # Remove the final newline stringvar=${stringvar%?} # prepend arr=("new_element1" "new_element2" "..." "new_elementN" "${arr[@]}") # Append to array arr=( "${arr[@]}" "new_element" ) myarray+=('foo') myarray+=("$line") # Append only if it's new containsElement() { local e for e in "${@:2}"; do [[ "$e" == "$1" ]] && return 0; done return 1 } if ! containElement "$line" "${myarray[@]}"; then myarray+=("$line") fi # Append to a specific index (e.g. at arr[2]) arr=( "${arr[@]:0:2}" "new_element" "${arr[@]:2}" ) # Remove an element at a specific index (e.g. arr[2]) arr=( "${arr[@]:0:2}" "${arr[@]:3}" ) # Length echo Number of elements in array: ${#myarray[@]} # Loop array for i in "${myarray[@]}"; do echo "$i" done
15.21.8.1. Associative array
declare -A mymap mymap[foo]=bar echo ${mymap[foo]} declare -A MYMAP=( [foo]=bar [baz]=quux [corge]=grault ) k=baz mymap[$k]=quux ${mymap[$k]} # is the same as ${mymap[baz]} Key with spaces can be double and single quoted but they are the same as not quoting them mymap[foo a]="bar b" mymap["foo a"]="bar b" mymap['foo a']="bar b" # Testing whether a value is missing from an associative array if [ ${MYMAP[foo]+_} ]; then echo "Found"; else echo "Not found"; fi # Found if [ ${MYMAP[missing]+_} ]; then echo "Found"; else echo "Not found"; fi # Not found declare -A MYMAP=( [foo a]=bar [baz b]=quux ) echo "${!MYMAP[@]}" # Print all keys - quoted, but quotes removed by echo # foo a baz b # Loop through all keys in an associative array for K in "${!MYMAP[@]}"; do echo $K; done # foo a # baz b # Loop through keys and values in an associative array for K in "${!MYMAP[@]}"; do echo $K --- ${MYMAP[$K]}; done # foo a --- bar #baz b --- quux # Redeclare, clear the whole array declare -A MYMAP MYMAP[foo]=bar echo ${MYMAP[foo]} # bar declare -A MYMAP # Re-declaring DOES NOT clear an associative array echo ${MYMAP[foo]} # bar unset MYMAP # You need to unset and re-declare to get a cleared associative array declare -A MYMAP echo ${MYMAP[foo]} # Unset a key MYMAP[foo]=bar echo ${MYMAP[foo]} # bar unset ${MYMAP[foo]} # WRONG echo ${MYMAP[foo]} # bar unset MYMAP[foo] # To delete from an associative array, use "unset" with similar syntax to assigning # BUT see next section if key contains spaces echo ${MYMAP[foo]} MYMAP[baz]=quux echo ${MYMAP[baz]} # quux K=baz unset MYMAP[$K] # Can unset using a variable for the key too # BUT see next section if variable may contain spaces - always better to quote echo ${MYMAP[baz]} # Unset a key that has spaces unset MYMAP[foo Z] # WRONG unset MYMAP["foo Z"] # You must quote keys containing spaces when you unset in an associative array echo ${MYMAP[foo Z]} # Length, same as indexed array declare -A MYMAP=( [foo a]=bar [baz b]=quux ) echo ${#MYMAP[@]} # Number of keys in an associative array
15.21.9. echo, print
# without quotes, need to escape special characters echo $o, world \(planet\)! # single quotes will not intepret variables echo '$o, world (planet)!' # $o, world (planet)! # enable backslash escape echo -e "Hello\nWorld"; # echo newline -e to use \n etc # always wrap variable in double quotes to preserve newline character
15.21.10. printf bash:printf
# e.g. Print multilines printf '%s\n%s\n' "$line1" "$line2"
15.21.11. Here document, here string bash:heredoc
The following is equivalent to
some-interactive-script < command-filesome-interactive-script <<EOF command #1 $varName EOF
- suppress leading tabs in the document body
- don't expand variables/command
- can be combined with
- <<-'EOF'
- can be combined with
Store as a variable
read -r -d '' VAR <<EOM This is line 1. This is line 2. Line 3. EOM echo "$VAR"
- http://tldp.org/LDP/abs/html/here-docs.html
- Here string
some-interactive-script <<<$myCommandorsome-interactive-script <<<"$myCommand"wc -w <<< "This is a test."grep "nor" <<<$var >/dev/null && echo "Found" || echo "Not found"equivalent to
echo $var | grep -q "nor" && echo "Found" || echo "Not found"
15.21.12. Read command output line by line, Prompt
15.21.12.1. Read command output line by line
multiple_commands_output="" output=$(drush command output contains multiple lines); # more examples output=$(docker stats --no-stream --format "{{.Name}}: {{.CPUPerc}}") # a file inputfile="/path/to/txt/file" count=1 while IFS=$'\n' read -r line; do echo "Line number $count: $line" count=$[ $count + 1 ] # refer to bash:heredoc done <<<"$output" # for a file # done <"$inputfile" multiple_commands_output="$multiple_commands_output""$output"$'\n' # You don't have to change back IFS as it only affects the while loop # newline character is \n # read -r :: to ignore all backslashes
15.21.12.2. Prompt
# `-n 1` accept one character # `-p` outputs a prompt # `-r` accepts backslashes literally otherwise it would see the backslash as an escape and wait for a second character # `$REPLY` is the default global variable the command uses to store the answer input # To specify a variable: read -p "..." -r my_var read -p "Are you sure? [Y/n]" -n 1 -r echo # (optional) move to a new line # `=~` to match a regex if [[ ! $REPLY =~ ^[Yy]$ ]] then exit 1 fi
15.21.13. Date format
date +%F- YYYY-MM-DD
date +%M- minute 00..59
date +%H- 00..23
date +%S- second 00..60
date +%T- HH:MM:SS
date +%Z- time zone e.g. EDT
date +%FT%T%Z- 2017-10-13T16:39:02UTC
date +%FT%H%M%S- 2017-10-13T163902 use this as it is filename safe
date +%FT%H%M%S%Z- 2017-10-13T163902EDT use this as it is filename safe
mydate=$(date +"%m-%d=%Y")- bash variable
15.21.14. Last executed command
#!/bin/bash set -o history -o histexpand var1='hello' echo $var1 echo !!
15.21.15. Passing arguments to command bash:pass arguments
It is ok to pass variables as argument value which is preceded by the argument name
ua='User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' curl https://a.ca/ -L \ -H "$ua"
Use bash:array to pass argument name and value
# create a non-associative array declare -a curlArgs=('-H' "keyheader: value" '-H' "2ndkeyheader: 2ndvalue") curl "${curlArgs[@]}" # args inside args declare -a historyDirs=(":(icase)wp-content/themes" ":(icase)wp-content/plugins") declare -a gitlogArgs=("--author=$j" '--since=last 1 month' '--pretty=tformat:' '--numstat' '--' "${historyDirs[@]}") git log "${gitlogArgs}"
15.21.16. String Operators
# A string is delimited by : # Get the last element foo=1:2:3:4:5 echo ${foo##*:} # '##' means greedy front trim # * matches any character # until the last ':' # also work for space as delimiter # ${foo##* } # Concatenate string variables z="" a="Hello\n" b="World\n" c="!" z="$z""$a" z="$z""$b" # insert a newline z="$z""$c"$'\n'
15.21.17. String comparison
# Start with 'node' if [[ $a == node* ]]; then # Start with a string 'Hello World' if [[ $a == "Hello World"* ]]; then
15.21.18. Numeric comparison
Not floating-point values
| Equal | $n1 -eq $n2 |
| greater or equal | $n1 -ge $n2 |
| $n1 -gt $n2 | |
| $n1 -le $n2 | |
| $n1 -lt $n2 | |
| not equal | $n1 -ne $n2 |
15.21.19. Bash: shell variable with default value
./yourcustom.sh var1 var2this_var1=$1 this_var2=${10} # maximum 11 variables this_var3=${3:-"_"} # if var3 is not provided, then default is string _
15.21.20. if bash:if
- Exit code zero then
conditionsis true ;&&&||
if [ conditions ]; then # commands elif [ conditions ]; then # more commands elif [ conditions ]; then # more commands else # more commands fi if [ -z "$(ls -lA)" ]; then echo "no files found" else echo "There're files" fi # if [ condition ] :: all POSIX shells # if [[ condition ]] :: new upgrad e.g. whether a string matches a regular expression. Supported by ksh, bash and zsh # if ((condition)) :: Supported by ksh extension, b ash and zsh. Return zero if the result of the arithmetic calculation is nonzero # if (command) :: Run a command in a subsehll # if command :: Run a command
! EXPRESSION- The EXPRESSION is false
-n STRING- The length of STRING is greater than zero
-z STRING- The lengh of STRING is zero (ie it is empty)
STRING1 = STRING2- STRING1 is equal to STRING2
- STRING1 != STRING2
- STRING1 is not equal to STRING2
INTEGER1 -eq INTEGER2- INTEGER1 is numerically equal to INTEGER2
INTEGER1 -gt INTEGER2- INTEGER1 is numerically greater than INTEGER2
INTEGER1 -lt INTEGER2- INTEGER1 is numerically less than INTEGER2
- -d FILE
- FILE exists and is a directory
-e "$myfile"- FILE exists
-f "$myfile"- File is a regular file (not a dir or device file)
- -r FILE
- FILE exists and the read permission is granted
-s "$myfile"- FILE exists and it's size is greater than zero (ie. it is not empty)
-w FILE- FILE exists and the write permission is granted
-x FILE- FILE exists and the execute permission is granted
15.21.21. Function bash:function
function_name () { } # or function function_name { } # Pass arguments print_something () { local var1='local var1' echo Hello $1 echo $var1 # local var1 # global var1 is not changed echo $var2 # global var2 return $1 } var1='global var1' var2='global var2' print_something Mars output=$( print_something "Mars" ) output=$( print_something "$1" )
15.21.22. Special Parameters
man bash- expands to
"$1" "$2" "$3" ... - expands to the name of the shell script file
$*for arg in $* do echo "Command line arg: $arg" done # ./run.sh One Two Three # Command line arg: One # Command line arg: Two # Command line arg: Three for arg in "$*" do echo "Command line arg: $arg" done # ./run.sh One Two Three # Command line arg: One Two Three
- number of arguments
- exit status of last run command
- process ID of the current shell
- the last command
- last argument of the last command
15.22. Cron
15.22.1. Install
apt install cron sudo systemctl enable cron sudo systemctl start cron sudo systemctl status cron
15.22.2. Explain https://crontab.guru/
15.22.3. Cron table
- Each user can have his own cron table. Use
crontabto manage the current user's cron jobs crontab -lcrontab -u username -l- All users' crontabs
- Each user
sudo ls -al /var/spool/cron/crontabsorsudo ls -al /var/spool/cron/
cat /etc/crontab- Cron jobs of packages usually saved under
/etc/cron.d/*use the same syntax as/etc/crontab/ - 4 cron directories. Copy script files to one of them and and they will be run as root
ls -al /etc/cron.*- /etc/cron.daily, /etc/cron.hourly, /etc/cron.montly, /etc/cron.weekly
crontab -l > ~/cron.backcrontab -e
echo $PATH may have to be copied to either the script file or crontab because env in cron is different from env the shell that you run the script in.
# Setup a Linux cron to run a Drupal Cron URL every 5 minutes */5 * * * * /path/to/rundrupalcron.sh
15.22.4. User crontab syntax and system-wide syntax
User Crontab Syntax min hour dayofmonth month dayofweek command
/var/spool/cron/crontabs/yourusernameor/var/spool/cron/*
System Crontab Syntax min hour dayofmonth month dayofweek user command
- /etc/crontab
range :: 1-5 wildcard :: *
dayofweek :: can be (mon, tue, wed, thu, fri, sat, sun) or (1, 2, 3, 4, 5, 6, 0)
# Last day of month 00 12 * * * if [ `date +%d -d tomorrow` = 01 ] ; then ; command # Every 15 minutes every Wednesday */15 * * * 3 echo "do something # Every hour except 6am 00 0-5,7-23 * * * echo "hi"
Special strings to replace with all 5 time-and-date fields
| @reboot | Run once, at startup. |
| @yearly | Run once a year, "0 0 1 1 *". |
| @annually | (same as @yearly) |
| @monthly | Run once a month, "0 0 1 * *". |
| @weekly | Run once a week, "0 0 * * 0". |
| @daily | Run once a day, "0 0 * * *". |
| @midnight | (same as @daily) |
| @hourly | Run once an hour, "0 * * * *". |
15.22.5. Sample
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games 0 2 * * * /home/li/cron/a.sh 10 2 * * * /home/li/cron/b.sh */20 0-2,6-23 * * * /home/li/cron/c.sh 10 3 * * * /home/li/cron/d.sh 10 19 * * 0 /home/li/cron/e.sh
15.22.6. Non cron way
Every 5 seconds
#!/bin/bash while true; do echo "hi" sleep 5 done
15.23. InCron - inotify cron
sudo apt-get install incron # /etc/, the following are created # - incron.allow # - incron.conf # - incrond.d/ # - incron.deny # one user per line e.g. root nano /etc/incron.allow # https://linux.die.net/man/8/incrond systemctl start incron.service # or /etc/init.d/incrond start incrontab -l incrontab -e
# <path/to/folder/or/file> <mask> <command> # Mask: https://linux.die.net/man/5/incrontab # IN_CLOSE_WRITE :: file opened for writing was closed. # IN_MODIFY :: file was modified. Before file open for writing is closed, there may be more than 1 time the file/folder is modified # Wildcards # $$ dollar sign # $@ watched filesystem path (see above) # $# event-related file name # $% event flags (textually) # $& event flags (numerically) # example /path/to/file IN_CLOSE_WRITE /usr/bin/php /home/vmail/ma-notify.php $@ $# $%
15.24. upx linux:upx
Executable file packer (compresser)
apt-get install upx-ucl # Most compressed level. Result file go.upx upx --brute go
15.25. Commands
15.25.1. xclip bash:xclip
sudo apt install xclip # select/copy xclip -sel clip < ~/.ssh/id_rsa.pub
15.25.2. screen bash:screen
It creates a screen session
- In a screen session, multiple shell windows (virtual terminals) can be open from a single SSH session.
- Processes running in Screen will continue to run when their window is not visible even if the SSH session is terminated
- If network connection fails in one screen session, screen will automatically detach the session.
- https://www.howtogeek.com/662422/how-to-use-linuxs-screen-command/
Installation
which screen sudo apt update sudo apt install screenC-a ?Commands about screen session
# open a screen session with a new window with a new shell screen # To create a screen session with a name screen -S my_session_name # Detach the current screen session C-a d # List running screen sessions screen -ls # Restore a detached screen session using session ID screen -r 10835 # simply quit or exit all windows in the attached session will close the session quit exit # To close a session # - -X: execute command # - -S: session PID screen -XS 10835 quit
Commands about windows in a screen session
# Create a window with shell inside a screen session # ~C-a c~ # Switch between windows # - next window ~C-a n~ # - previous window ~C-a p~ # - Go to a specific window ~C-a 0~ ~C-a 1~
Commands about regions in a screen session
# To split a window into 2 regions: the first region is the current window, the second region is empty # - Split the window horizontally: ~C-a S-s~ # - Split the window vertically: ~C-a |~ # Move cursor to the next region (the empty region) # ~C-a Tab~ # Display a window in a new region # - choose to open the first window: ~C-a 0~ # - Or create a new window: ~C-a c~ # Region navigation # - Go to next region: ~C-a~ # - Close all regions except the current one: ~C-a q~ # - Close the current region: ~C-a x~
- Get alert
- To get an alert when there's output registered on a specific screen, on that screen run
Ctrl-a Mand then switch to other screen - Or get an alert when there's no output
C-a _
- To get an alert when there's output registered on a specific screen, on that screen run
C-a xC-a korexitfor each screen window.screen -lsC-a "it's double quoteS-'C-a A
Run a background process after SSH signout
screen nohup ./main & # C-a d exit
15.25.3. Run Sequentially
lspci; lsusb
If the first command not successful, 2nd command will not run: lspci && lspci
15.25.4. ls
List files or directories by name
ll- Ubuntu server default
ls -alF
- Options
- sort by time last modified
-t- sort by file size
-S- reverse sort
- r
- in detail
- l
- In MB
-l --block-size=M- Append indicator to entries
- -F, –classify
- @
- symbolic link (or that the file has extended attributes)
- *
- executable
- =
- socket
- |
- named pipe
- >
- door
- /
- directory
- Output immediately
-1- do not sort; list entries in directory order
-U- do not sort, enable
-aU, disable -ls –color -f
- Results of
ls -l-rwxr-xr-x 1 10490 floppy 17242 May 8 2013 acroread- First
-represents a regular file. It could be- d
- directory
- c
- character device
- l
- symlink
- p
- named pipe
- s
- socket
- b
- block device
- D
- door
- -
- regular file
- Large directory
ll -U -1- do not sort, print immediately
ll -1 -f- do not sort, print immediately without color
ls -l my_scr?pt
ls -l my*
ls -l my_s*t
ls -l my_scr[ai]t
ls -l f[a-i]ll
ls -l f[!a]ll
15.25.4.1. colors
- Bash shell
eval $(echo "no:global default;fi:normal file;di:directory;ln:symbolic link;pi:named pipe;so:socket;do:door;bd:block device;cd:character device;or:orphan symlink;mi:missing file;su:set uid;sg:set gid;tw:sticky other writable;ow:other writable;st:sticky;ex:executable;"|sed -e 's/:/="/g; s/\;/"\n/g') { IFS=: for i in $LS_COLORS do echo -e "\e[${i#*=}m$( x=${i%=*}; [ "${!x}" ] && echo "${!x}" || echo "$x" )\e[m" done }
15.25.5. less
less afile.txt # show line numbers at the footer less -M afile.txt # show line numbers on the left less -N afile.txt
- Go back and up one page
fb- Go to start and end of the file
gS-g- exit
q
15.25.6. head, tail
- Last 5 lines
tail -n5 /path/to/file- Keep tailing
tail -f /var/log/messages- Used in chain
ls -t | tail -n3- exit
C-c
15.25.7. man
- Search
processesin command names and description man -k processes- Search in command names and title
man -f ls- Save manual to a txt file
man ls > ls.txt
15.25.8. cd - Navigate to previous directory (back)
15.25.9. Word Count wc
wc filename- Output
13 14 123 filename- 13 lines, 14 words, 123 characters
- Line Count e.g.
ps -ef | wc -l - Word Count
- Byte Count
- Character Count
15.25.10. sort
sort filename- output sorted lines. File is not changed
- Options
-f, --ignore-case-r, --reverse-u, --unique
15.25.11. uniq
uniq filename- Different from linux:sort, uniq dedup consecutive lines
uniq -d filename- Find lines that appear at least twice in consecutive order
uniq -u filename- Filter out lines that repeated consecutively
15.25.12. Replace String, Stream Editor sed
- Replace all (g)
NickwithJohncase-insensitivecat report.txt | sed 's/Nick/John/gi' > report_new.txt
sed SCRIPT INPUTFILE...- or
sed OPTIONS... [SCRIPT] [INPUTFILE...]
- or
- SCRIPT
scommand- substitution
's/{regexp}/{replacement}/{flags}'- To literally match string, wrap each letter in
[and]and for^do[\^]
- e.g. literal string
'' sed 's/[&][#][x][1][f][;]//gi' -i c.xml- literal string
'&#^x1f;' sed 's/[&][#][\^][x][1][f][;]//gi' -i c.xml- (no term)
- Special characters to escape for
{regexp}- Single quote
- \'
$\$.\.*\*[\[\\\]\]^\^`\`
- (no term)
{replacement}- (no term)
- e.g.
echo 'daytime' | sed -E 's/(...)time/\1light/', result is ~xxxlight- Up to
\9
- Up to
- (no term)
- Use
&to subsitute the whole matched portion of the pattern - (no term)
- Special characters to escape for
{replacement}- Single quote
\'- (no term)
&and\need to be escaped, as do the delimiter (usually/) and newlines\n
- (no term)
- Refer to regexp and replacement escape
- literal string
- (no term)
- Instead of
/as delimiter, use|for handling string with/such ashttp://- Always use
\to escape no matter what delimiter is
- Always use
- (no term)
- For inside bracket expression (list of characters), to literally include:
^- put it not at the beginning
[ab^c]. If it's the only one, use[\^] -- put it either at the start or end
[-abc]or[abc-]not[a-bc]. If it's the only one, use[\-] ]- put it at the start
[]abc]or[^]abc]but not[abc]]nor[abc\]]. If it's the only one,[^]]
- Notice
- escape
\is crucial because\numbere.g.\1is subsitution,\lettere.g.\nalso has special meaning - Notice
- use double quotes for interpolation
sed -e "s/$BRE/$REPL/"
- OPTIONS
-E- extended regex
-e SCRIPT- run several inline SCRIPT's described above
-f SCRIPT-FILE- Run commands in SCRIPT-FILE's
-i- files are edited in-place. Original files are overwritten, no backup files are created
- Mac
cavat
sed -i "s/ENCRYPTION='Y'//g" ../../../mysql-ma-bk/backup.sql # sed: 1: "../../../mysql-ma-bk/ba ...": invalid command code . # Should be sed -i '' -e "s/ENCRYPTION='Y'//g" ../../../mysql-ma-bk/backup.sql
- (no term)
- For deleting only
sed "s/deletethis//g" -i input.txtsed 's///gi' -i c.xml
-i.backupor-i .backup(only Mac)- copy original to
*.backupand change original files- For multiple files
- No backup
sed 's/Nick/John/gi' *.txt- create backup files
sed 's/Nick/John/gi' -i.backup *.txt- (no term)
- ::
- For multiple files
- (no term)
-n, --quiet, --silent# Assume the file has # The quick brown fox jumps over the lazy dog. # Notice :: there're 2 substitution commands here sed -e 's/fox/coati/;s/dog/dingo/' myfile # prints output after all commands # The quick brown coati jumps over the lazy dingo. # suppress output sed -n -e 's/fox/coati/;s/dog/dingo/' myfile # suppress output but print result after the first command sed -n -e 's/fox/coati/p;s/dog/dingo/' myfile # The quick brown coati jumps over the lazy dog. # p can be a standalone command. These 2 commands are equivalent sed -n -e 's/fox/coati/p;s/dog/dingo/' myfile sed -n -e 's/fox/coati/;p;s/dog/dingo/' myfile # These 2 commands print out the final output sed -n -e 's/fox/coati/;s/dog/dingo/;p' myfile sed -n -e 's/fox/coati/;s/dog/dingo/p' myfile
- man sed
- Refer usage in mysql:unknown collation
15.25.13. xargs
- Build and execute commands from standard input. It converts input from standard input into arguments to a command
echo 'lorem.txt' | xargs wc # output whole command before run (-t) echo 'lorem.txt' | xargs -t wc # loop # `-n 1` means use at most 1 argument per command line echo 'a.txt' 'b.txt' | xargs -t -n1 wc # eq. to wc a.txt wc b.txt # Similarly, `-L 1` means use at most 1 nonblank input line per command line cat a.txt | xargs -t -L1 wc # placeholder cat fruit.txt | xargs -I :FRUIT: echo "buy more: :FRUIT:" # only delimit input on newline but not space, `-0` # e.g. there're some directories or files that have spaces in names ls ~/Library/ | grep 'A.*' | xargs -0 -n1 # combine with find's -print0 find test1/ -type f -print0 | xargs -0 chmod 755 # interactive, `-p` find . -name "*.backup" -depth 1 -print0 | xargs -p -0 rm # find total number of lines of all files in current directory find . -name '*.php' | xargs wc -l
15.25.14. test - Conditionally run
# -L :: if the file exists and it's a symbolic link test -L /bin/systemctl || ln -sf /usr/bin/systemctl2 /bin/systemctl
15.25.15. Concatenate String with File
# Append to the beginning # - echo returns a newline echo '<?xml version="1.0"?>' | cat - input.txt > output.txt # '-' after 'cat' means to take the stdin from last command # Append to the end echo 'end line' | cat input.txt - > output.txt
15.25.16. Change Encoding
- Change from UCS-2 to UTF-8
iconv -f UCS-2 -t UTF-8 input.xml > output.xml
15.25.17. Comment out every line in a file and save as a new file
Add '# ' in front of every line jail.conf is the source file, and jail.local is the destination file. After it's run, jail.conf is not affected.
awk '{ printf "# "; print; }' /etc/fail2ban/jail.conf | sudo tee /etc/fail2ban/jail.local
15.25.18. sudo
Install sudo apt-get update && apt-get install -y sudo less
Run sudo interactive sudo -i
15.25.19. SHA, salt hashing image:lucee:salt
./luceehashing.sh UIloginPassword salt
#!/bin/bash SHA_ALGORITHM=256 SHA_COUNT=5 LUCEE_PASSWORD=${1:-"_"} LUCEE_SALT=${2:-"_"} if [ $LUCEE_PASSWORD == "_" ]; then LUCEE_PASSWORD=topsecret fi if [ $LUCEE_SALT == "_" ]; then LUCEE_SALT=$(uuidgen | tr a-z A-Z) fi COUNT=1 LUCEE_HASH=$(echo -n "${LUCEE_PASSWORD}:${LUCEE_SALT}" | shasum -a $SHA_ALGORITHM | cut -f1 -d' ') while [ $COUNT -lt $SHA_COUNT ]; do LUCEE_HASH=$(echo -n $LUCEE_HASH | shasum -a $SHA_ALGORITHM | cut -f1 -d' ') COUNT=$((COUNT + 1)) done echo "Lucee Admin Values" echo " hspw = $LUCEE_HASH" echo " salt = $LUCEE_SALT"
15.25.20. byobu
Create multiple terminals and switch between.
apt-get install byobu
15.25.21. Command line web browser w3m
Ubuntu
apt-get install w3m # S-Q to quit
15.25.22. set bash:set
# shell set -ex; command1; command2; ... # bash script set -ex command1 command2
- -x
- enable a mode of the shell where all executed commands are printed to the terminal
- -e
- under common situation, exit when there's an error (stop at first error)
15.25.23. Calendar
cal- show current month
cal 12 2020- show December, 2020
cal -y 2020- show all months in 2020
cal -y- current year
15.26. Cheat
sudo apt-get update && sudo apt-get upgrade sudo apt-get install python-pip sudo pip install cheat cheat -v
Modify ~/.bashrc
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
export EDITOR="/usr/bin/nano"
export CHEATCOLORS=true
# don't put duplicate lines or lines starting with space in the history.
HISTCONTROL=ignoreboth
Add to autocompletion
cd /etc/bash_completion.d/
# sudo wget https://raw.githubusercontent.com/chrisallenlane/cheat/master/cheat/autocompletion/cheat.bash
# cheat.bash is as follows
function _cheat_autocomplete {
sheets=$(cheat -l | cut -d' ' -f1)
COMPREPLY=()
if [ $COMP_CWORD = 1 ]; then
COMPREPLY=(`compgen -W "$sheets" -- $2`)
fi
}
complete -F _cheat_autocomplete cheat
cheat -d lists the directories that cheat sheets should be saved. One is ~/.cheat, the other is system
Each cheat sheet is a file, like ~/.cheat/ping
;; # some comments ping -c 15 www.example.com ;; # another usage ping ...
cheat -s packets search all cheat sheets that has `packets` in commands and description.
Also return cheat sheet name
cheat ping show the `ping` cheat sheet
15.27. Makefile
- Install in Ubuntu
sudo apt-get update && sudo apt-get install make- Docs
- https://www.gnu.org/software/make/manual/make.html
Create a Makefile in current folder
include env_make # include a file which defines some variables NAME=myimagename VERSION=0.1.0 REPO=mydockerreponame # define some variables # escape $ in string variable with $$, say password is $abc$ password=$$abc$$ # trylogin: # sshpass -p '$(password)' ssh -o StrictHostKeyChecking=no root@123.123.123.123 # Define dynamic variable filename := file_$(shell date +%FT%T%Z).log .PHONY: default build # This tells make to not look elsewhere for commands default and build # Instead, run default and build as they are defined in this Makefile # Usually include all commands in this Makefile # And all file or directory names that exist in the directory that Makefile is in # command in Makefile is called target. Syntax of Makefile is # target ...: prerequisites ... # recipe # ... # ... # A prerequisite is a file/command. It means to include a file # A recipe is an action (command). Every recipe line has to start with a tab character default: @echo "Some description" # If you run `make` in current folder without specifying which command, the first command will be run # echo without @ prints # echo "Some description" # while @echo prints # Some description all: build # the all command runs the build command build: sudo docker build -t $(NAME):$(VERSION) --rm . push: docker push $(NAME)/$(REPO):$(VERSION) release: build make push -e VERSION=$(VERSION) # `make release` runs `build` first, then run `push` with a parameter # can't put push as a prerequisite after `build` because it has to be run using a parameter testecho: @echo $$FOO # make testecho FOO="a b c" # actually @echo $(foo) then ~make testecho foo="a b c"~ testsub: @echo otheraction $(filter-out $@,$(MAKECMDGOALS)) # make testsub a b c # $@ is the target name, which is testsub # $(MAKECMDGOALS) is the list of targets, which is testsub a b c # This method requires you to put these 2 lines to the end of Makefile (remove leading #'s) # %: # @: # # %: is a target which matches every target # @: is a recipe in which @ is the same as @echo. : means do nothing # Escaping $ might be needed for running sub command start: docker rm -f $$(docker ps -a -q) # - in front of command means ignore the exit status of the command so that a non-zero exit would not stop the recipe runrecipeinmiddle: - echo do something - $(MAKE) build testpwd: cd ~/abc; echo `pwd`; # ~/abc echo $(PWD); # the path where `make testpwd` is run
- Makefile
make your-recipe- Makefile.test
make -f Makefile.test your-recipe- Makefile.live
make -f Makefile.live your-recipe
Be careful to close single quotes in @echo
15.27.1. Characters to escape
https://www.cmcrossroads.com/article/gnu-make-escaping-walk-wild-side
Escape # as \# Escape $ as $$
15.27.2. Options
15.27.2.1. -C dir, --directory=dir
phpcs: $(MAKE) --directory api phpcs # make -f ./api/Makefile phpcs
15.28. Nano
- Meta key (M-) from Mac is
Escape - C-o
- C-x
- M-u
- M-e
- C-c
- Show line number when open
nano -c filename
- C-u
C-wor F6, type string and enter to search,Alt + Wto search nextC-w C-vorC-_ C-v- Select and copy, paste
S-M-aorC-6to set mark, move cursor to selctS-M-6orM-6to copy or useC-kto cut textC-uto pasteTo delete all content in a file
- Go to the beginning of the file, set mark
C-6 - Go to the end of the file
C-w C-v
- or
S-M-t - but Zoom may take over this shortcut and do screenshot instead. Close Zoom to avoid its shortcut
- Delete
Backspace
- Delete
- Go to the beginning of the file, set mark
Esc, release, then $ (shift+4)orM-S-4C-S-chave to ensure visual word wrap is on for multiple linesM-S-q- https://nano-editor.org/dist/latest/cheatsheet.html
- MacOS iTerm2
- Enter Copy Mode
⇧ ⌘ c- Toggle selection by character
Space- (no term)
- Move cursor to select
- Copy
y- Stop selecting
C-Space- Paste
⌘ v
15.28.1. TS: File is being edited by root
SSH connection failed/interrupted when editing a file, nano saved a snapshot
FILE.txt.saveor.FILE.txt.swp- Delete the snapshot/lock file
15.29. Vim
- Interactive tutorial
- https://www.openvim.com/
15.29.1. Normal Mode
15.29.1.1. Move Cursor
- h, j, k, l
- start of next word
- end of the word
- beginning of the word (previous word beginng)
- {number}{h, j, k, l, w, e, b}
15.29.1.2. Copy, Cut, Paste
dd- delete a line and add it to the default register
p- paste below the current lines
S-p- paste above the current line
ddkP- move up a line
ddp- move down a line
15.29.1.3. Replace
ciw- change inner word. Delete word at cursor and go to Insert Mode
15.29.1.4. :cq :: exit with an error and the file is not saved
15.29.2. Insert Mode
30i- Esc- insert
-30 times and jump back to normal mode
15.30. Ubuntu
15.30.1. Gnome
- Logout
gnome-session-quit- Logout gnome session in SSH remote session (haven't used it myself)
env DISPLAY=:0.0 gnome-session-quit --logout --force- Reboot in SSH remot session
env DISPLAY=:0.0 gnome-seesion-quit --reboot
15.30.2. Shortcuts
- Most of system shortcuts
- Settings > Devices > Keyboard
- Search a shortcut containing
s gsettings list-recursively | grep -i \>s\'- (no term)
- e.g. it returns this
org.gnome.shell.extensions.screenshot-window-sizer cycle-screenshot-sizes ['<Alt><Control>s'] - (no term)
sudo apt install dconf-editor && dconf-editorand go to that folder- (no term)
- Turn off
Use default valueand in Custom value change it from['<Alt><Control>s']to[] - Terminal
C-M-t- Copy from Terminal (GNOME)
C-S-c- Open favorite app 1
Super-1- Show All Applications
Super-a- Show Notification tray
Super-m- Run commands
M-<F2>- (no term)
- Switch workspaces
C-M-UporC-M-Down - (no term)
- Switch apps (open App Switcher)
M-Tab - (no term)
- Switch instances/windows for the current app
M-` - Quit app in App Switcher
M-q- Close application window
C-qorM-F4- (no term)
- Drag and drop with
C-Sto create symlinks
15.30.3. Setting
https://linuxconfig.org/things-to-do-after-installing-ubuntu-18-04-bionic-beaver-linux
- Keyboard repeat rate
Settings > Universal Access > Typing > Repeat Keys > On, lower speed faster rate
- Remote Desktop from Windows RDP
http://c-nergy.be/products.html
chmod +x ~/Downloads/Std-Xrdp-Install-0.5.1.sh ./Std-Xrdp-Install-0.5.1.sh -s yes -g yes sudo reboot # try remote from Windows
- Install Pinyin
sudo apt install ibus-pinyin- Settings > Languages and Region > Manage installed Languages, reboot or select IBus as Keyboard input method system, add Chinese Simplified then reboot
- Settings > Languages and Region > add Chinese (Intelligent Pinyin) as a new Input Sources
- Software & Updates > Ubuntu Software > Download from > Other > Select Best Server
gnome-tweak-toolsudo add-apt-repository universesudo apt install gnome-tweak-tool- Open
Tweaksapp
- prepare to install extensions that come with the package repo
- Log back in and go to Tweak Tool > Extensions
- GDebi
- By default, Ubuntu Software Center does not install dependencies when app .deb is installed
sudo apt install gdebi- e.g. download deb from Chrome and in Files app, right click > Properties > Open with GDebi Package Installer, close
sudo gdebi google-chrome-stable_current_amd64.deb- To remove a package installed from gdebi,
aptanddpkgcommands usingpurgeoption as shown:sudo apt purge google-chrome-stablesudo apt-get purge teamviewersudo dpkg -purge teamviewer
- Snap
- Snap is built-in in Ubuntu 18.04
- Apps installed by Snap are almost independent. Different versions of the same app can co-exist
- Before developers have to come up packages for each distro and each version of distro. Now, one snap one distro.
- Snap Daemon can be used in distros other than Ubuntu
- Snaps are automatically updated (4 times a day)
- Snap is installed globally for all users once and each user stores separate userdata
- https://snapcraft.io/store
sudo apt install snapd- Find Snap package
snap find vlc sudo snap install vlcsnap listsnap changes- Upgrade
sudo snap refresh vlc sudo snap refresh --list- Rever a recent update
sudo snap revert vlc sudo snap remove vlc- stable, candidate, beta, edge
sudo snap refresh vlc --channel=edge
- Download and install it offline later
snap download vlc- download .assert and .snap file
- (no term)
snap ack vlc.assert- (no term)
snap install vlc.snap
sudo apt install ubuntu-restricted-extras- Intel graphic card screen tearing under X11
sudo mkdir -p /etc/X11/xorg.conf.d/ sudo nano /etc/X11/xorg.conf.d/20-intel.conf
Section "Device" Identifier "Intel Graphics" Driver "intel" Option "TearFree" "true" EndSection
Reboot
- Shutter linux:shutter
sudo apt install shutter- https://launchpad.net/ubuntu/+archive/primary/+files/libgoocanvas-common_1.0.0-1_all.deb
- https://launchpad.net/ubuntu/+archive/primary/+files/libgoocanvas3_1.0.0-1_amd64.deb
- https://launchpad.net/ubuntu/+archive/primary/+files/libgoo-canvas-perl_0.06-2ubuntu3_amd64.deb
sudo apt install libappindicator-devsudo cpan -i Gtk2::AppIndicatorsudo killall shutter- Start Shutter again
- Pixelize, Delay & Selection
- Files app (Nautilus)
- Right click create new file (New Document > test.txt)
touch ~/Templates/test.txt
- Check for updates
ubuntu-support-status
15.30.4. Dual Boot with Win 10
- Turn off fast start up
- Control Panel > Hardware and Sound > Power Options > Choose what the power buttons do > uncheck Turn on fast startup
- May need to disable Secure Boot
- enable means BIOS will prevent un-authorized OS be loaded.
- Win + S to search
advanced startup optionsand clickRestart Now - select TroubleShoot > Advanced Options > UEFI Firmware Option Settings > Restart
- BIOS settings will open, disable Secure Boot. You may need to Turn Legacy Support On/Off
- Win + S to search
- (no term)
- Determine if it's BIOS or EFI mode
- Run
msinfo32and see if BIOS Mode is UEFI- BIOS mode
- boosts by reading the first sector on hard disk and executing it; this boot sector in turn locates and runs additioinal code. It uses the Master Boot Record (MBR) partition table which is very limiting because of space (no more than 2TB in size per partition) and partitions(more than 4 primary partitions) constraints.
- EFI mode
- boots by loading EFI program files (with .efi extension) from a partition on the hard disk uses the GUID partition table (GPT) offering 64-bit entries in its table which dramatically extends the support for size possibilites.
- Ubuntu has to be installed using the same mode. If you see black screen with options when you boot using Ubuntu's bootable USB, then it shows Ubuntu is booted from EFI.
- Run
- Create free space on Windows for Ubuntu
- Win + S to search
Create and Format Hard Disk Partitions> Shrink Partition, so that you will haveUnallocatedspace in partitions. You will need 2 unallocated spaces. One for root and one for /home in Ubuntu - (no term)
- Create a bootable USB stick on Windows
- Boot from USB
- Win + S to search
Advanced Startup Options> Restart Now > Boot from USB - (no term)
- Try Ubuntu without installing > open gparted, right click on unallocated to create partition
- Create as: Primary Partition, File system: ext4, Label: ROOT and HOME, partition name: root and home,
- (no term)
- Open Install on the desktop
- (no term)
- Normal installation and Install third-party software for graphics and Wi-Fi hardware, MP3 and other media
- Installation type
- Something else
- change on a device name
- Use as: Ext4 journaling file system, Mount point: / and /home, Format the partition.
- Device for boot loader installation
- If boot mode is BIOS, select the whole device name e.g.
/dev/sda. If it's UEFI mode, choose the partition. Go togpartedand see which partition has valuebootin column 'Flags'. e.g./dev/sdap1
15.30.5. Remmina to Win 10
- Enable RDP on Win 10
- Remina change connection
Color depthtoGFX RFXorHigh color (16 bpp) - By default, use Right Ctrl to toggle
Grab all keyboardand then Win key can be passed to Win 10- Remina > Preferences > Keyboard > Grab keyboard
15.30.6. Citrix
- Better to install Workspace app for Linux (Full Package 64bit) instead of Citrix Receiver for Linux
TS: Cannot connect to "Your Connection Name" No such file or directory. Verify your connection settings and try again.
cd /opt/Citrix/ICAClient/keystore/
sudo cp -ar cacerts cacerts-bk
sudo rm -rf cacerts
sudo ln -s /etc/ssl/certs cacerts
15.30.7. Keychain
- Search password to get
Passwords and Keys - App usually use
Passwords > Login - phpStorm uses
IntelliJ Platform DB — hash-id-get-it-from-proj-folder/idea/dataSources.xml
15.30.8. Sync date time
# 18.04 # see if System clock synchronized is yes timedatectl status # check time difference between the local system and the internet time server, and sync if necessary sudo chronyd -q # to install sudo apt install chrony
15.30.9. Maintenance
sudo apt autoremove --dry-run- delete any unused dependencies
sudo du -sh /var/cache/apt- APT keeps previously downloaded and installed packages even after they've been uninstalled
- Remove only the outdated packages
sudo apt autoclean --dry-run - Clean it entirely
sudo apt clean --dry-run
- Remove only the outdated packages
15.31. Ubuntu Personal Package Archive - PPA
- Add a LibreOffice package from PPA to Software Sources
sudo add-apt-repository ppa:libreoffice/ppa, thensudo apt-get update- Remove the package source
sudo add-apt-repository --remove ppa:libreoffice/ppa,apt-get update- Remove it and downgrade back to previous version before using PPA
sudo apt-get install ppa-purge,sudo ppa-purge ppa:libreoffice/ppa
15.32. Production Server Setup
- linux:user
- Add a normal user, add the user to sudo group
- linux:ssh
- Add ssh public key to the new user's .ssh folder
- linux:ssh:disallow_root
- Disallow SSH in as root
- linux:sudo:nopassword
- require no password for non-root users with sudo privilige/group
- linux:firewall
- configure firewall, open ports
- linux:timezone
- setup timezone and sync time
16. Domain Server, DNS Server, SSL/https
See linux:dns
16.1. DNS records: A AAAA ANAME CNAME TXT ALIAS URL
- Hostname
- a sub domain. e.g.
www@ A record- redirect hostname.domainname.com to an IP address. For the barebone domain name, the hostname is @
- wildcard
- hostname
*or*.asubdomain
AAAArecords- same as A records but they are IPv6. Nowadays, both AAAA and A are needed
ANAME- It's like
CNAME. Name (from source domain) could be @. The value has to be another (target) apex domain. DNS resolves the target fully qualified domain name (FQDN) to an IP or multiple and also synthesize A records of the domain that points to the IP(s) of the FQDN. Since the target FQDN's IP is returned on the first lookup, it's faster than ALIAS record. CNAMErecords- point another subdomain (hostname) to an
Arecord (e.g. hostname.domainname.com). It should only be used when there are no other records on that name. Don't use the bare domain inCNAMEon the left hand side. ALIASrecords- same as
CNAMEbut it can coexist with other records on that name. It resolves the target domain to one or more A records. ALIAS's name can be (from source domain) anything, the value can be (target) any other external domain. URLrecords- redirect the name to the target name using HTTP 301 status code
- It redirects the name to a destination. The
URLrecord is simple and effective way to apply a redirect for a name to another name, e.g. to redirectwww.example.comtoexample.com
- It redirects the name to a destination. The
TXTrecord- Google Search Console may add TXT @ google-site-verification=[code] TTL 1 Hour to verify site ownership.
A,CNAME,ALIASrecordsAname must resolve to an IP, theCNAMEandALIASrecord must point to a name- (no term)
- General rule
- use an A record if you manage what IP addresses are assigned to a particular machine or if the IP are fixed (this is the most common case)
- use a CNAME record if you want to alias a name to another name, and you don't need other records (such as MX records for emails) for the same name
- use an ALIAS record if you are trying to alias the root domain (apex zone) or if you need other records for the same name
- use the URL record if you want the name to redirect (change address) instead of resolving to a destination.
- (no term)
- On bare/apex domain, usually A, ALIAS or ANAME record can be used. Only DNS Made Easy supports ANAME and DNSimple and others support ALIAS. ANAME and ALIAS are not industry standards
16.2. MX record dns:mx
- HOSTNAME
- e.g. use @ for barebone
- Mail Providers Mail Server (or Value)
- always end with "." e.g.
alt4.aspmx.l.google.com. - Priority
- the smaller the more important
Google G Suite requires TTL to be 1 hour (3600). Priority might be different but aspmx.l.google.com. must be the highest
Priority Mail Server
1 ASPMX.L.GOOGLE.COM.
5 ALT1.ASPMX.L.GOOGLE.COM.
5 ALT2.ASPMX.L.GOOGLE.COM.
10 ALT3.ASPMX.L.GOOGLE.COM.
10 ALT4.ASPMX.L.GOOGLE.COM.
16.3. SPF record dns:spf
SPF record is a TXT record which has 0 or more ordered mechanisms
all | ip4 | ip6 | a | mx | ptr | exists | include
Mechanism can be prefixed with 1 of 4 qualifiers. A qualifier means if the machanism matches the sender's IP, then return a result. If a mechanism matches the sender but no qualifier, then + is used. If no mechanism or modifier matches, then default result is Neutral.
+ Pass - Fail ~ SoftFail ? Neutral
# Google G Suite SPF record v=spf1 include:_spf.google.com ~all # Add domain's A, MX records to the match list and always allow. # Include a domain to the match list. # Soft fail for all other senders IP or domains. v=spf1 +a +mx include:_spf.google.com ip4:1.2.3.4 ip4:1.2.3.5 ip4:1.2.3.6 ~all
- Mechanism
allmatches all senders' IP's, and it's usually at the end:-allfail for all~allsoft fail for all+allallows all senders
- Mechanism
mxadds domain's MX details to the match list. - Mechanism
aadds domain's A records to the match list
16.4. DKIM record dns:dkim
- One domain can have multiple DKIM records
- Mail server signs the email with the private key and the receiving mail server uses the public key in the domain's DNS info to verify the signature
Name: s1._domainkey Value: s1.domainkey.xxx.xx.sendgrid.net- On the mail server e.g. SendGrid
TXT s1._domainkey the-domain-key-valuesee below
TXT customer._domainkey k=rsa; p=xxxxfor a root domain, for subdomainTXT customer._domainkey.e k=rsa; p=xxxx- When the email is sent from an SMTP mail server e.g. SendGrid, the receiver will have Sender Domain (the From: email address), Envelope Domain (e.g. Zoho Campaign mail server is zcsend.net) and DKIM Domain
- Envelope Domain is the domain used to receive bounce error messages (e.g. bounce rate) from recipient systems and I think it's specified in the sent email as a header
Return-Path - DKIM Domain is specified as a header in the sent email using DKIM-Signature
v=1; a=rsa-sha256;d=abc.com;s=customer;c=relaxed/relaxed;q=dns/txt;t=1554220118;h=x-fbl:mime-version:date:message-id:content-type:list-unsubscribe:from:to:subject; bh=XXXHAHSED; b=XXXHAHSED- the sign algorithm
- the signing domain which is also the DKIM Domain
- the selector used to find DKIM public key info. To match the domain's DKIM record e.g. as in
customer._domainkey - the canonical algorithm(s) for header and body
- the default query method
- signature timestamp
- the list of signed header fields, repeated for fields that occur multiple times. Basically saying this signature verifies these fields are true
- body hash
- the actual digital signature of the contents (headers and body) of the mail message
- If Sender Domain and Envelope Domain do not match, then SPF is not aligned. If Envelope Domain is a subdomain of Sender Domain, they are aligned
- If dns:dmarc has
aspf=rwill reject if SPF is not aligned. But it's common to have SPF not aligned - Next is to check if DKIM Domain is aligned with Sender Domain. If they don't match, DMARC will definitely fail. So you will have to have a different DKIM record setup for each Sender Domain
16.5. SOA record
The SOA record stores information about:
- the name of the server that supplied the data for the zone;
- the administrator of the zone; e.g. b.c.com is actually the email address b@c.com
- the current version of the data file;
- the number of seconds a secondary name server should wait before checking for updates;
- the number of seconds a secondary name server should wait before retrying a failed zone transfer;
- the maximum number of seconds that a secondary name server can use data before it must either be refreshed or expire;
- and a default number of seconds for the time-to-live file on resource records.
A DNS zone is the part of a domain for which an individual DNS server is responsible. Each zone contains a single SOA record.
e.g. ns2.a.com. b.c.com. 2018012804 10800 3600 604800 10800
16.6. redirect.center, redirect.name
If URL record is not available, use CNAME and redirect.center and redirect.name Instructions: https://milanaryal.com/domain-redirecting-and-url-forwarding-with-a-simple-dns-record/
redirect.center is especially good for redirect subdomain www.abc.com to google.com/path if the domain host doesn't support Domain Forwarding with path.
To redirect naked domain, use Forward Domain.
;; redirect google.example.com to google.com
google.example.com IN CNAME google.com.redirect.center
;; redirect google.example.com to google.com with a 302 status code
google.example.com IN CNAME google.com.opts-statuscode-302.redirect.center
;; If a user try to visit www.oldwebsite.com/about/ will redirect to path www.newwebsite.com/about/:
www.oldwebsite.com IN CNAME www.newwebsite.com.opts-uri.redirect.center
;; opts-statuscode-{code} :: HTTP Status Code to be used in the redirect. 302, HTTP Status Code
;; opts-uri :: Append URI (if any) to the target URL
;; opts-slash
;; jobs.my-domain.com to http://www.my-domain.com/jobs
;; jobs IN CNAME to www.my-domain.com.opts-slash.jobs.redirect.center
;; multiple path components, jobs.my-domain.com to www.my-domain.com/jobs/xyz
;; jobs IN CNAME to www.my-domain.com.opts-slash.jobs.opts-slash.xyz.redirect.center
;; Use redirect.name instead of readirect.center
;; redirect google.example.com to google.com
google.example.com IN CNAME alias.redirect.name
_redirect.google.example.com IN TXT Redirects to https://www.google.com
;; redirect google.example.com to google.com with options
google.example.com IN CNAME alias.redirect.name
_redirect.google.example.com IN TXT Redirects from /* to https://www.google.com/*
;; Redirects to [target], where target is the target URL
;; Redirects from [path] to [target], where path is a path to match on the hostname
;; Redirects permanently to [target], where permanently redirects with a 301 status code (defaults to 302 otherwise)
;; redirect www.my-domain.com to https://my-domain.com
www.my-domain.com IN CNAME my-domain.com.opts-https.redirect.center
16.7. Domain Forwarding
Naked domain forwarding Forward abc.com to http://google.com abc.com/xyz will become google.com/xyz in the address bar
Forward abc.com to http://google.com/xyz Some domain host and registration service might support that. abc.com/ijk will become google.com/xyz/ijk in the address bar
Subdomain forwarding can be done as well.
Domain registrar might require to purchase SSL with domain in order to forward https://olddomain.com to https://newdomain.com
Mask is http://oldomain.com continues to appear as oldomain.com in address bar but it actually points to newdomain.com
16.8. Domain Transfer
- Make sure domain Admin contact email is valid. Unlock domain and remove privacy. Ask for auth code
- Go to new registrar, in order to maintain the old nameservers, GoDaddy requires to setup DNS for the domain before request for domain transfer
- Purchase domain transfer and enter auth code. .CA domains can be transferred immediately
- On new registrar, create a DNS template and input old DNS records
- At the end, change the nameservers to the new registrar and apply the DNS template with old records
16.9. Addon domain
It's an addtional domain that the system stores as a subdomain abc.com is registered in GoDaddy and will be added as an addon domain on abc.xyz.com. Change nameservers to the domain xyz.com host say ns.googlehosting.com ns2.googlehosting.com Then in googlehosting, set it up as an addon domain :: abc.com > subdomain abc.xyz.com, then specify the document root
abc.com will appear as parked on subdomain abc.xyz.com on the googlehosting
An addon domain has to be mapped to a main domain's subdomain. In this case, the subdomain is abc.xyz.com
Source domain is abc.com registered in GoDaddy and nameservers are later changed to destination domain's Google nameservers. Destination domain is abc.xyz.com and the root domain xyz.com is registered and DNS managed in Google.
You don't need to first create a subdomain and then create Addon domain. It will ask you to create a subdomain in the destination domain and point to a directory when you create an addon domain.
Source domain can be a subdomain e.g. s1.abc.com, change A record for s1.abc.com at GoDaddy to point to the destination's server IP.
16.10. Park domain
- To park a domain means to make the domain as idle. Nameservers will be set to Default e.g. in GoDaddy
- To park multiple domains
abc.net,abc.infoto the main domainabc.com
16.11. Domain Registrars and Tools
- GoDaddy
Delegate Access godaddy:delegate access
- Source GoDaddy account
s@a.cagives limited access to another GoDaddy accountd@a.ca
- Source account > Account Settings > Delegate Access > choose to send to
b@a.ca - The person clicks an invite in mailbox
b@a.ca, then logs on tod@a.ca. Then GoDaddy accountd@a.cacan access GoDaddy accounts@a.ca
- Source GoDaddy account
- Hover.com
- search for the right domain using your words
16.12. DNS History, IP location, Website Hosting Finder
- DNS history
- https://securitytrails.com/dns-trails
- See all DNS records
- https://www.ultratools.com/tools/dnsLookup
- Lookup domain registrar and nameservers use whois
- https://mxtoolbox.com/Whois.aspx
- Find IP location
- https://www.iplocation.net/
- DNS propagation
- https://www.whatsmydns.net/
- (no term)
- Find hosting of a website
16.13. AXFR and Find subdomains
Without DNS Zone file export, it's almost impossible to find all subdomains and their DNS records. The following can be used:
- Brute force
dig a.x.comdig b.x.com - Pentest-tools.com - Find Subdomains">not guaranteed to show all subdomains
- If AXFR is enabled on DNS server and the IP from which this command is run is added as a secondary IP address to the target domain, the run
host -l x.comordig @ns1.DomainsSOA.com x.com axfr - Use Google search results to filter out known subdomains
site:x.com -"www.x.com" -"a.x.com" -"b.x.com"
DNS servers host are known as zones. DNS zone transfer is to replicate db changes across several DNS servers. AXFR is referred as DNS zone transfer because it's the protocol used during a DNS zone transfer.
Initiate an AXFR zone-transfer request is simple and can be used by hackers. The below replicate x.com DNS records to the first nameserver.
dig @ns1.DomainSOA.com x.com axfr
16.14. AutoSSL in WHM/cPanel autossl:whm
- WHM and cPanel provides free SSL certificates for VPS and Dedicated server users
- It uses SHA256 TLS v1 and expires every 90 days. TLS v1 must be disabled for eCommerce site by June 30, 2018. For non-eCommerce site, it's ok
- AutoSSL requires Domain Control Validation (DCV) - the domain has to have NS or A records point to the server that AutoSSL is generated. Otherwise, it will not do anything, not even creating
.well-knownfolder - First is to enable AutoSSL for WHM users. AutoSSL will check all root domains under the user account
- If you enable a user and the user has 1 root domain and several addon domains and subdomains, all will have SSL
- AutoSSL will include certificates for www. domains. e.g. www.yourrootdomain.com, www.yoursub.yourrootdomain.com
- yoursub.yourrootdomain.com was set up by you and you didn't setup www.yoursub.yourrootdomain.com
- Use hosting nameservers in domain registrar. Log in WHM as root. Go to Manage AutoSSL.
- Providers
- cPanel (powered by Comodo)
- Options
- Check all options.
- "Allow AutoSSL to replace invalid or expiring non-AutoSSL certificates"
- enable this to allow AutoSSL to replace certs that the AutoSSL system did not issue e.g. you import purchased cert. Enable this will overwrite the imported cert with AutoSSL certificates.
- Manage Users
- Enable AutoSSL for relevant users
- (no term)
- Click on Run AutoSSL for All Users
- (no term)
- Check Logs
- SSL certificates should be now installed
- http://www.inmotionhosting.com/support/website/cpanel/add-delete-autossl-cpanel
- A directory
.well-knownwill be created in sub website directory if the parent website directory has AutoSSL
16.15. Install SSL
- See SSL Test Tools
- certificate starts with
--BEGIN CERTIFICATE--, file extension can be .pem, .crt- Fingerprint of a certificate
- SHA-256
openssl x509 -noout -fingerprint -sha256 -inform pem -in your.pem- SHA-1
openssl x509 -noout -fingerprint -sha1 -inform pem -in your.pem- MD5
openssl x509 -noout -fingerprint -md5 -inform pem -in your.pem
- Fingerprint of a certificate
- private key starts with
--BEGIN RSA PRIVATE KEY-- - Certificate Chain or Chain of trust: end-user, intermediate and root certificate
- Certificate Authorities issue SSL/digital certificate
- certify the ownership of a public key (CSR) which is generated locally
- Install SSL certificate issued from CA1, this is called the end-user certificate
- this certificate basically vouches for the CSR which is a URL and its registration info
- CA1 utilizes a certificate issued by CA2 and CA2 utilizes a certificate issued by CA3
- CA2's certificate is an intermediate CA
- CA3 is a root CA and its certificate is directly embeded in the web browser and is called root certificate since its CA is trusted by the browser
- Certificate Authorities issue SSL/digital certificate
Generate a private key
myserver.keywith 2048 bits, then generate a CSR (Certificate Signing Request) filemyserver.csrusing that private key and answering some registration questions. The CSR contains the answers linux:ssl:server.csr.cnf and a public keyopenssl req –new –newkey rsa:2048 –nodes –keyout myserver.key –out myserver.csr # Prompt for Common Name # example.com for a single domain # *.example.com for wildcard certificate
- Submit CSR to SSL authority (CA) to get
.crtand usemyserver.keygenerated above and make them root user only - Generate self-signed certificate
docker-compose.yml
ports: - "32012:80" - "443:443"
- Create a Root SSL Certificate for domain
localhostopenssl req -newkey rsa:4096 -nodes -keyout rootCA.key -x509 -days 5000 -out rootCA.pem
- create a new private key and also certificate request (CR)
-x509- create a self signed certificate rather than a cerficiate request (CR)
-nodes- create private key without providing passphrase which is at least 4 letters long
- (no term)
- Enter random values for the prompt
- Optionaly, you can create a certificate with an existing key
- (no term)
openssl req -new -nodes -key rootCA.key -x509 -days 1024 -out rootCA.pem- (no term)
-newwith-keycreates a CR with an existing key- Combine private key and certificate into a .p12 (pkcs12, pfx format) file
- (no term)
openssl pkcs12 -inkey rootCA.key -in rootCA.pem -export -out rootCA.p12- (no term)
- Validate
openssl pkcs12 -in rootCA.p12 -noout -info- Add
rootCA.p12if Chrome supports it. Windows can import .p12 file
- Add
- (no term)
- On Chrome, Settings > Manage certificates > Import under Trusted Root Certification Authorities
- (no term)
- After that, select the certificate with name Internet Widigits Pty Ltd. click Advanced and select all Certificate purposes
- (no term)
- In Ubuntu, Chrome only supports pkcs7 (.p7b file). So just add
rootCA.pemto Chrome as Authorities - (no term)
- Without adding to Chrome,
https://localhostonly works in InCognito mode.
Now create domain certificate using the root SSL certificate
- Create
server.csr.cnfso you don't have to type linux:ssl:server.csr.cnf
[req] default_bits = 4096 prompt = no default_md = sha256 distinguished_name = dn [dn] C=US ST=RandomState L=RandomCity O=RandomOrganization OU=RandomOrganizationUnit emailAddress=hello@example.com CN = localhost
CN is common name which is a FQDN
- Create
v3.extfor creating X509 v3 certificate
authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = localhost
- Now create a certificate key
server.keyfor localhost and also create a certificate signing request (CSR)server.csrusingserver.csr.cnf
openssl req -new -nodes -out server.csr -newkey rsa:4096 -keyout server.key -config <( cat server.csr.cnf )- Genereate the domain certificate file
server.crtusingv3.extthat is created earlier
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 5000 -extfile v3.ext- Some read and verify commands
# Read and verify a certificate openssl x509 -text -noout -in server.crt # Read and verify a key openssl rsa -check -in server.key # Read and verify a CSR openssl req -text -noout -verify -in server.scr # Read and verify a PKCS#12 file .pfx or .p12 openssl pkcs12 -info -in rootCA.p12 -noout
- Create
- Combine certificates and install
- Nginx requires to package the intermediate certificates in a single bundle with the end-user certificate
Apache requires to bundle the intermediate certificates and assign the location of the bundle to
SSLCertificateChainFileconfig- Inside the PHP + Apache container, copy the
<VirtualHost: *:80>and make it*:443and add 3 lines of SSL config as described in apache:ssl
a2enmod ssl apache2ctl configtest apache2ctl graceful
- Redirect HTTP to HTTPS
- apache:https
- Inside the PHP + Apache container, copy the
- header:strict-transport-security
16.16. SSL Testing Tools
- https://www.whynopadlock.com/
- Qualys SSL Lab https://www.ssllabs.com/ssltest/
17. Docker
17.1. Version, Info, Installation
- Brief
docker --version- Server and Client
sudo docker version- Full
sudo docker info- (no term)
- "Docker for Windows" needs Windows Hyper-V enabled. Command prompt is admin
- (no term)
- After installation, make sure linux:firewall DEFAULT_FORWARD_POLICY="ACCEPT"
- (no term)
- Use
systemctlfor managing Docker service
17.1.1. Without sudo
Users in user group docker can run docker commands without sudo.
# replace your username with ${USER} sudo usermod -aG docker ${USER} # to apply the new group membership, you can log out and back in, or run this su - ${USER} # print name (-n, --name) instead of a number and show group IDs (-G) id -nG
17.1.2. Install on Digital Ocean
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04
sudo apt update # Add GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # Add Docker repo to APT sources so that Docker is the most up to date sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt-get update # query the APT cache, it will show docker-ce is from the docker repo apt-cache policy docker-ce sudo apt-get install -y docker-ce # check if Docker is running sudo systemctl status docker
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04
sudo apt update # install prerequisite packages which let apt use packages over HTTPS sudo apt install apt-transport-https ca-certificates curl software-properties-common # add GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # add Docker repository to APT sources: sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" # make sure to install Docker from the Docker repo instead of the default Ubuntu repo: apt-cache policy docker-ce sudo apt install docker-ce # check status sudo systemctl status docker # add your username to docker group to avoid typing sudo sudo usermod -aG docker ${USER} # logout or type the following to apply the new group membership su - ${USER} # confirm your username is now added to the docker group id -nG
Install docker:docker compose:ubuntu
17.2. Command Help docker docker-subcommand --help
17.3. Images
- List all images
docker images- List all dangling images which can be removed
docker images -f "dangling=true"- Remove all dangling images (-q is to return IDs only)
docker rmi $(docker images -f "dangling=true" -q --no-trunc)- Remove all none images
docker rmi $(docker images | grep "none" | awk '/ / { print $3 }')- Download an image
docker pull phusion/baseimageor with a tagdocker pull phusion/baseimage:0.9.18docker pull --platform=linux/amd64 mysql:5.7.40
- Remove an image
docker rmi imageName- Save an image to tar file
docker save -o /path/docker-image-file.tar docker-image-name- Import a tar to image
docker load -i /path/docker-image-file.tar- Search image in Docker repository
docker search ubuntu- Setting for an image
docker image inspect php:7.0-fpm
Extract Dockerfile from an image. Use one or more of below
- Use image:dockerfile-from-image
docker image inspect <image-name>docker history --no-trunc <image-name>sudo ./test.sh <image-name>
#!/bin/bash docker history --no-trunc "$1" | \ sed -n -e 's,.*/bin/sh -c #(nop) \(MAINTAINER .*[^ ]\) *0 B,\1,p' | \ head -1 docker inspect --format='{{range $e := .Config.Env}} ENV {{$e}} {{end}}{{range $e,$v := .Config.ExposedPorts}} EXPOSE {{$e}} {{end}}{{range $e,$v := .Config.Volumes}} VOLUME {{$e}} {{end}}{{with .Config.User}}USER {{.}}{{end}} {{with .Config.WorkingDir}}WORKDIR {{.}}{{end}} {{with .Config.Entrypoint}}ENTRYPOINT {{json .}}{{end}} {{with .Config.Cmd}}CMD {{json .}}{{end}} {{with .Config .OnBuild}}ONBUILD {{json .}}{{end}}' "$1"
17.3.1. docker build, docker tag
- With one Dockerfile in a folder (resource). Run to build
docker build -t your-image-name .- (no term)
Rename image: Change
myimage(:latest)tomyimage:1.0docker tag myimage myimage:1.0 # rename from myname1 to myname2 docker image tag myname1 myname2 # docker image tag eq. docker tag docker rmi myname1
docker tag myimage myimage:1.0
- Build an image and tag it with latest and 2.0
docker build -t myimage:latest -t myimage:2.0 .- Multiple Dockerfiles (Dockerfile1, Dockerfile2) in the current folder
docker build -t your-image-name -f Dockerfile2 .- Dockerfile and resource are in different folders (e.g. COPY a file from parent folder in Dockerfile)
docker build -t your-image-name -f /path/Dockerfile.yourname /path/to/docker/- (no term)
Build a lot of times while testing a Dockerfile
docker build --rm --force-rm --no-cache -t my-image-name .
17.4. run
17.4.1. Interactive mode with tty -it
If there's an error, cannot enable tty mode on non tty input, try -i only
17.4.2. Run an image in a container and keep running in background
docker run --name wordpressdb -v=/vagrant/mydbdump:/tmp/mydbdump \ -v=/mydbdata:/var/lib/mysql \ -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=wordpress \ -d mysql:5.7.9
Container name: wordpressdb Image name: mysql:5.7.9 Detached mode (in background): -d Insert environment variables: -e
17.4.3. Run an image in a temp container only once, then remove the container
A MySQL container, wordpressdb, created by mysql:5.7.9 image, is already running.
Start up another container using the same image In this temp container, wordpressdb container can be discovered as wpdb (alias). Then import a .sql file to wpdb The temp container is then removed.
Refer to docker:link
docker run -it --rm \ -v=/vagrant/mydbdump:/tmp/mydbdump \ --link wordpressdb:wpdb mysql:5.7.9 \ sh -c \ 'exec mysql -h"$WPDB_PORT_3306_TCP_ADDR" \ -P"$WPDB_PORT_3306_TCP_PORT" \ -uroot -p"$WPDB_ENV_MYSQL_ROOT_PASSWORD" \ wordpress < /tmp/mydbdump/pantheon_db_sql'
17.4.4. Link docker:link
Deprecated and it relies on Docker in default bridge network mode.
17.4.5. –name and –hostname
Specify the container name using –name
Inside the container, the container is called –hostname=value
docker run --hostname=value OR docker run -h value
Combine –name and –hostname as much as you can
17.4.6. –privileged docker:run:privileged
e.g. allow docker container to run docker daemon, allow docker container to access all devices such as file systems.
17.4.7. Port -p 1234:3000
17.4.8. --platform linux/x86_64
- See dockerfile:from:platform
- Try to use
--platform linux/amd64instead oflinux/x86_64 - When an image is run, may encounter this warning. Having the platform option changes the host platform arch
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
17.4.9. -v, --volume vs --mount volume
- https://docs.docker.com/storage/volumes/
- Bind mount vs named volume
- Bind mount
- no volume is created, direct map host path to container path
- Named volume or data volume
- fully controlled by Docker (Linux VM)
- See 17.8
- See 17.11.3.3
- Dockerfile and docker-compose can use relative path in source. However,
docker runcannot. Use linux:makefile For named volumes, both
-vand--mountcover all options# -v volumeName:/target/in/container:comma-separated,list,of,options,eg,ro docker run -d --name devtest -v myvol2:/app nginx:latest # --mount # type=bind, volume or tmpfs # source= or src= # destination= or dst= or target= # docker run -d --name devtest --mount source=myvol2,target=/app nginx:latest
17.5. Containers cp logs stats start stop restart rm inspect
- List containers
docker ps -a- Stop and Start
docker stop containerNamedocker start containerName- Restart
docker restart containerName- Stop and remove
docker rm -f containerName- Stop and remove all containers
docker rm -f $(docker ps -a -q)- Stop and remove all exited containers
docker rm $(docker ps -qa --no-trunc --filter "status=exited")- Container log
docker logs -f containerName-f to follow log output
17.5.1. docker inspect
docker inspect containerName- docker:inspect:ip">
NetworkSetting > IPAddressor
docker inspect containername | grep -i "IPAddress"
- Volume
Config > VolumesandMounts- Return true or false for container running state
docker inspect -f {{.State.Running}} $CONTAINER_ID
17.5.2. docker events - Show docker system log
17.5.3. docker cp
- Copy files from container
ghostto docker host docker cp -L ghost:/usr/src/ghost/config.js ./config.js- Copy files from docker host to container
docker cp -L ./config.js ghost:/usr/src/ghost/config.js- Copy a folder, result
/var/www/transfer/web docker cp web/. ghost:/var/www/transfer/web- Copy a folder, result
/var/www/transfer/web docker cp web/ ghost:/var/www/transfer- -L
- copy symbol link
- -a
- copy file permissions as well as userid:groupid
SRC_PATH specifies a file
- DEST_PATH does not exist the file is saved to a file created at DEST_PATH
- DEST_PATH does not exist and ends with / Error condition: the destination directory must exist.
- DEST_PATH exists and is a file the destination is overwritten with the source file’s contents
- DEST_PATH exists and is a directory the file is copied into this directory using the basename from SRC_PATH
SRC_PATH specifies a directory
- DEST_PATH does not exist DEST_PATH is created as a directory and the contents of the source directory are copied into this directory
- DEST_PATH exists and is a file Error condition: cannot copy a directory to a file
- DEST_PATH exists and is a directory
- SRC_PATH does not end with /. (that is: slash followed by dot) the source directory is copied into this directory
- SRC_PATH does end with /. (that is: slash followed by dot) the content of the source directory is copied into this directory
17.5.4. docker stats
- CPU and Memory usage for all containers
docker stats -adefault just shows running containers
Show container names
docker stats --format "table {{.Name}}\t{{.Container}}\t{{.CPUPerc}}\t{{.MemPerc}}\t{{.MemUsage}}\t{{.PIDs}}" docker stats --no-stream --format "{{.Name}}: {{.CPUPerc}}" # other column values # \t{{.NetIO}}\t{{.BlockIO}}
17.5.5. Save a running container to an image
First exit the container first
docker commit -m "added node.js" -a "Author Name" 123456 my/ubuntu-nodejs
Container id :: 123456. get this from docker ps -a
New image name :: my/ubuntu-nodejs
17.5.6. Execute commands inside a running container
- Login to container with bash open
docker exec -it containerName bash -I- (no term)
- Long command (pure-pw only exists in container)
docker exec -it containerName pure-pw useradd bob2 -f longpathto/passwd -m -u ftpuser -d /home/ftpusers/bob2
- Run multiple commands
docker exec -it containerName bash -c 'cd /var/log; tar -cv ./file.log'
17.6. Network
;; # List networks
docker network ls
;; # Info about a network
docker network inspect the_network_id
;; # Remove networks
;; # returns error when last bridge is removed. It's normal
docker network rm $(docker network ls | grep "bridge" | awk '/ / { print $1 }')
network has active end points. Find the Containers that has Name (mycontainer_name) and EndpointID
docker network inspect mynetwork_name ;; # you may need to stop and remove the container docker rm -f mycontainer_name ;; # disconnect the container from a network docker network disconnect -f mynetwork_name mycontainer_name ;; # run docker-compose down again docker-compose down -v
17.7. My Network
Refer to CIDR Netmask network:port Personal dev projects use 172.20.x.0/24 Work dev projects use 172.19.x.0/24
Personal dev project port range up to 49151
- mysql
- 31000 to 31999
- php
- 32000 to 32999
Personal dev project
- 172.20.0.0/24
- wordpress, port mysql:31000 php:32000
- 172.20.1.0/24
- drupal 7, port mysql:31001 php:32001
networks:
app_netsandbox:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.20.0.0/24
gateway: 172.20.0.1
services:
dbtest:
networks:
app_netsandbox:
ipv4_address: 172.20.0.2
17.8. Volume
17.8.1. Basics
- See docker-compose:volumes
- See 17.4.9
# List all volumes docker volume ls # Info about a volume docker volume inspect the_vol_id # a volume can be created # - when using `docker run -v some_volume:/path/to/container` # - when using docker-compose # - when using docker volume docker volume create --name dbdata # Remove volumes dangling docker volume rm $(docker volume ls -qf dangling=true) # Remove a volume docker volume rm dbdatavolume
17.8.2. Access the named volume by starting another container
On Windows, you may be forced to have named volume because file permissions cannot be set. In this case, you can start a new container to access the volume
Container yourContainerName has named volume dbdata pointed to /var/www/html
# check what files are in the volume docker run -ti --rm --volumes-from yourContainerName ubuntu:latest ll /var/www/html # backup volume `dbdata` to host backup/backup.bar docker run --rm --volumes-from yourContainerName -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /var/www/html # In the backup.tar, the file structure is var/www/html/* # restore volume for a container # create a container `bkcontainer` with `dbdata` and bash in docker run -v /dbdata --name bkcontainer ubuntu /bin/bash docker run --rm --volumes-from bkcontainer -v $(pwd):/backup ubuntu bash -c "cd /dbdata && tar xvf /backup/backup.tar --strip 1"
17.8.3. File System Sharing
- Before,
osxfs~/~gRPC-FUSE+Hypervisor Frameworkwas used on Mac, nowVirtioFS+Virtualization Frameworkis used- See 17.11.3.3
17.9. Hosts file
- Cannot modify/override
/etc/hostsfile when building an image - Ways to modify
/etc/hosts- Change when running the image
- Use docker-compose
- Use
docker run
- Go into the running container and manually modify the hosts file
- Change when running the image
17.10. Dockerfile
17.10.1. FROM
- There're 3 ways to specify the target platform. Choose one that you like. For example, building an image on ARM64 and make it AMD64
- Set a global environment on docker host for building a single image using docker:build and Dockerfile
export DOCKER_DEFAULT_PLATFORM=linux/amd64
- In Dockerfile
- In docker-compose.yml where image is built docker:compose:platform
- Set a global environment on docker host for building a single image using docker:build and Dockerfile
- When an image is run, may encounter this warning
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested- refer to docker:run:platform for a warning
# x86 version FROM --platform=linux/x86_64 ubuntu:18.04
17.10.2. CMD, RUN, ENTRYPOINT
17.10.2.1. CMD
CMDis to provide defaults for a container that is already created from an image and running.CMDis just to provide paremeters toENTRYPOINTwhendocker runis rundocker run ...longoptions $image $other_command$other_commandoverridesCMD!
- There can only be one
CMDin a Dockerfile or a chain of Dockerfiles included as in parent images - See 4
- If there're multiple, only the last one will run
CMDdoesn't execute anything at build time and it's not committed to the image.ENTRYPOINTDocker's default entrypoint is to start a shell/bin/sh -c- Docker allows you to specify a command in
docker run, and that's to combine with entrypoint to run- e.g.
docker run -it ubuntu <my command>will run/bin/sh -c <my command> - where
<my command>could bebash
- e.g.
- Combine ENTRYPOINT,
docker run without extra commandand CMDENTRYPOINT ["redis", "-H", "something", "-u", "toto"]CMD ["get", "key"]docker run redisimgis eq. todocker run redisimg get key
- Docker allows you to specify a command in
- Specifying
entrypointindocker-compose.ymloverrides dockerfile'sENTRYPOINTand clears out dockerfile'sCMD
There're 3 forms of CMD
# exec form, json array, doesn't have variable substitution CMD ["executable", "param1", "param2"] # But you can be clever, this works CMD [ "sh", "-c", "echo $HOME" ] # as default parameters to ENTRYPOINT CMD ["param1","param2"] # shell form CMD echo $HOME
17.10.2.2. RUN
RUNruns a command and commits the result to an image at build time- Can be multiple RUN directives
- It runs using a shell
- Say dockerfile:shell is
/bin/sh -candRUN <my command>, it actually runs/bin/sh -c <my command>
RUN ["executable", "param1", "param2"]RUN <command>https://stackoverflow.com/questions/25899912/how-to-install-nvm-in-docker/60137919#60137919# change shell or load .bashrc # because non-interactive-non-login shell does not load .bashrc RUN /bin/bash -c 'source $HOME/.bashrc; \ echo $HOME'
17.10.2.3. SHELL
- Change default shell
- can be multiple
SHELLdirectives
- can be multiple
SHELL ["executable", "parameters"]["/bin/sh", "-c"]- Ubuntu is
["/bin/bash", "-c"]
["cmd", "/S", "/C"]
17.10.2.4. /bin/bash vs /bin/sh
- Most of the time
bashis an advanced shell that is on top ofsh - Sometimes
shis linked tobash - Alpine doesn't have
/bin/bash
17.10.3. ADD vs COPY
COPY <src>… <dest> COPY ["<src>",… "<dest>"] (this form is required for paths containing whitespace)
ADD and COPY are the same except
- ADD allows src to be an URL
- If the src is an archive (tar) it will be unpacked
Always use COPY if it can do what you want
17.10.4. Timezone
Works for Ubuntu
ENV TZ=America/Toronto RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
# /etc/localtime -> /usr/share/zoneinfo/Etc/UTC ll /etc/localtime cat /etc/timezone Etc/UTC
- See MySQL Docker image environment variables
17.10.5. ARG Debian (Jessie)
- Problem
- When building an image using Debian, a lot of these:
debconf: delaying package configuration, since apt-utils is not installed
- (no term)
- It's like
ENVbutENValways overridesARG - (no term)
- Define in Dockerfile without a value
- So that users can pass at build-time to the builder with the docker build command
--build-arg <varname>=<value>FROM busybox ARG user1 ARG buildno
- (no term)
Define
ARGafterFROMARG VERSION=latest FROM busybox:$VERSION ARG VERSION RUN echo $VERSION > image_version
ARG defined before FROM is outside of build stage so it can't be used in any other instruction
- (no term)
Define with a value
ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y apt-utils # eq. to RUN DEBIAN_FRONTEND=noninteractive apt-get ...
17.10.5.1. Predefined ARG variables
Docker has a set of predefined ARG variables without default values. Use it like without defining ARG in Dockerfile
docker build -t mycontainer . --build-arg <varname>=<value> --build-arg HTTP_PROXY=http://user:pass@proxy.lon.example.com
These predefined ARG variables are not saved in docker history so it's good to use them for passing sensitive info HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy NO_PROXY no_proxy
17.11. Docker Compose
17.11.1. Basics
docker-compose --version docker-compose up -d docker-compose up -d --force-recreate # Force to recreate containers docker-compose build --no-cache # Force to rebuild image for services defined in docker-compose.yml with build directive, and then you need to docker-compose up later # Use a different docker-compose file docker-compose -f docker-compose.prod.yml up -d docker-compose down -v # remove containers and volumes docker-compose down ---rmi local # remove only images that don't have a custom tag set by the `image` field docker-compose down --rmi all # remove all images used by any service docker-compose build # build new image if there's any build directive docker-compose up -d # doesn't build image, it will use existing image docker-compose up -d --build # build new image and run docker-compose ps # shows containers info started by docker compose in current folder
17.11.2. Docker Compose Ubuntu Installation docker:docker compose:ubuntu
Same for 16.04 and 18.04 Grab latest docker-compose version say it's 1.16.1
sudo curl -o /usr/local/bin/docker-compose -L "https://github.com/docker/compose/releases/download/1.16.1/docker-compose-$(uname -s)-$(uname -m)" # add execute permission sudo chmod +x /usr/local/bin/docker-compose docker-compose -v # Test nano docker-compose.yml my-test: image: hello-world # run docker-compose up docker rm <container-id> docker rmi hello-world
17.11.3. YML
17.11.3.1. version (top level)
- Compose and Docker Engine compatibility matrix
- https://docs.docker.com/compose/compose-file/
- Version 3 Doc
- https://docs.docker.com/compose/compose-file/compose-file-v3/
version: '2'
17.11.3.3. volumes
- See docker:volume
- Top Level
version: "3.9" services: web: image: nginx:alpine volumes: # Long syntax. Volume #1 # https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference - type: volume # first encounter of named volume `mydata`, and it's initialized # because `mydata` will be reused in other service `db`, so a top-level volume is needed source: mydata target: /data volume: # flag to disable copying of data from a container when a volume is created nocopy: true # Long syntax. Volume #2 - type: bind source: ./static target: /opt/app/static # Short syntax. Volume #3. data-volume needs to be at top-level - data-volume:/var/lib/mysql db: image: postgres:latest volumes: - "/var/run/postgres/postgres.sock:/var/run/postgres/postgres.sock" - "dbdata:/var/lib/postgresql/data" # Top Level volumes: # Named volume which can be used in all services # this volume isn't mapped to anything in docker host! # top-level volumes can't specify path to named volume.. You have to install other Docker plugin mydata: dbdata: data-volume:
- Short syntax
- Modes
- https://docker-docs.netlify.app/docker-for-mac/osxfs-caching/#tuning-with-consistent-cached-and-delegated-configurations
- Has no effect on old gRPC-FUSE e.g. (osxfs)
# Syntax [SOURCE:]TARGET[:MODE] # modes (comma separated) # - ro (read only), rw (read and write) # - consistent (default) :: perfect consistency (host and container have an identical view of the mount at all times) # - delegated :: the container’s view is authoritative (permit delays before updates on the container appear in the host). Writes performed by containers may not be immediately reflected on the host file system. In situations such as NFS asynchronous mode, if a running container with a delegated bind mount crashes, then writes may be lost. # - cached :: the host’s view is authoritative (permit delays before updates on the host appear in the container). Writes performed by containers are immediately visible to the host, but there may be a delay before writes performed on the host are visible within containers. volumes: # Just specify a path and let the Engine create a volume - /var/lib/mysql # Specify an absolute path mapping - /opt/data:/var/lib/mysql # Path on the host, relative to the Compose file - ./cache:/tmp/cache # User-relative path - ~/configs:/etc/configs/:ro # Named volume - datavolume:/var/lib/mysql
17.11.3.4. networks (top level) links (deprecated)
Every time docker-compose up creates a single network (per app) which all services in the app are on.
Let's say yml file is a folder called foldername
This avoids the need to use links
networks:
;; # network name is foldername_app_net
app_net:
;; # bridge on a single host (default) and overlay on a Swarm
driver: bridge
ipam:
driver: default
config:
- subnet: 172.18.0.0/16
gateway: 172.18.0.1
Then in a service, specify the ip
services:
db:
image: mysql:5.7.9
container_name: db
hostname: db
;; # links is deprecated.
;; # links:
;; # - thisserviceorotherservicename:mysql
;; # Use networks:aliases below for thisservice
networks:
app_net:
ipv4_address: 172.18.0.2
aliases:
- mysql
restart: always
- Use network defined in another docker-compose
;; # ./app1/docker-compose.yml networks: app_net: driver: bridge ipam: driver: default config: - subnet: 172.16.2.0/24 gateway: 172.16.2.1 services: lucee: ... networks: app_net: ipv4_address: 172.16.2.3 ;; # ./app2/docker-compose.yml networks: app1_app_net: external: true services: nodejs: ... networks: app1_app_net: ipv4_address: 172.16.2.4
17.11.3.5. services
- Basics
services: lucee: build: . # use Dockerfile to build an image image: cflucee:latest # specify image name which can be later used in other service # When there's build in yml, you need to add --build # docker-compose up -d --build container_name: lucee ports: - "80:80" - "8888:8888" volumes: - /host/path/to/folder1:/root/db/ - /host/path/to/folder2:/var/log/nginx # equivalent to -t in docker run tty: true networks: app_net: ipv4_address: 172.18.0.3
- ports
EXPOSEin Dockerfile and-pindocker run -p 8080- Neither specify EXPOSE nor -p
- service in container will not be accessible from anywhere except from the container itself
- Only specify EXPOSE
- The service in the container is not accessible from outside Docker, but from inside other Docker containers.
- Good for inter-container communication
- Specify EXPOSE and -p
- The service in the container is accessible from anywhere, even outside Docker
- Specify -p but not EXPOSE
- It's like only specify EXPOSE
- Neither specify EXPOSE nor -p
- user
- https://docs.docker.com/engine/reference/run/#user
- one line
- Both user/group name and user/group ID can be used
clickhouse: image: yandex/clickhouse-server ports: - "8123:8123" user: "101:101"
17.11.3.6. Resources
cpu_count: 2 cpu_percent: 50 cpus: 0.5 cpu_shares: 73 cpu_quota: 50000 cpuset: 0,1 user: postgresql working_dir: /code domainname: foo.com hostname: foo ipc: host mac_address: 02:42:ac:11:65:43 mem_limit: 536870912 #512 MB 512*1024*1024, 128MB 134217728 memswap_limit: 2000000000 mem_reservation: 512m privileged: true oom_score_adj: 500 oom_kill_disable: true read_only: true shm_size: 64M stdin_open: true tty: true
compose file v3
version: '3' services: redis: image: redis:alpine deploy: resources: limits: cpus: '0.50' memory: 50M reservations: cpus: '0.25' memory: 20M
17.11.3.7. Pass arguments from Compose to Dockerfile
docker-compose.yml
version: '2'
services:
web:
build:
context: ./web
args:
REQUIREMENTS: "envs/dev.env"
Dockerfile
FROM python:3.5 ENV PYTHONUNBUFFERED 1 ENV APP_ROOT /usr/src/app ARG REQUIREMENTS ... COPY $REQUIREMENTS $APP_ROOT/
17.12. Docker for Windows
Settings > General > Expose daemon on tcp://localhost:2375 without TLS. This is to allow legacy clients/applications such as phpStorm to connect to the daemon.
17.12.1. PowerShell Auto Complete
Win + x > PowerShell (Admin) Run these
;; # To allow downloaded scripts signed by trusted publishers to run on this computer
Set-ExecutionPolicy RemoteSigned
;; # Type y, check if this return RemoteSigned
get-executionpolicy
;; # Install PowerShell module posh-docker, it may also install NuGet package manager
Install-Module posh-docker
;; # Enable autocompletion for the current PowerShell only
Import-Module post-docker
;; # To make it persistent across all PowerShell sessions, we need to import the module to ~$PROFILE~
;; # Run this directly on PowerShell to create a profile if not exists and add the line
if (-Not (Test-Path $PROFILE)) {
New-Item $PROFILE –Type File –Force
}
Add-Content $PROFILE "`nImport-Module posh-docker"
;; # check
cat $PROFILE
17.13. Docker Cloud
Docker Cloud uses Docker Hub as native registry.
Use Docker Hub credentials to docker login
Assume your Docker Hub username is abc, give a tag for your image
docker tag myimage:mytag abc/myimage:mytag
docker push abc/myimage:mytag
docker logout
17.14. Docker registry
# docker login [OPTIONS] [SERVER] # Options # - --password, -p # - --password-stdin # - --isername, -u docker login my.dockerregistry.com docker login localhost:8080 # push a local image to registry # First, tag the local image with the host name or IP address, and the port of the registry docker tag myimage:latest my.dockerregistry.com/myimage:latest docker push my.dockerregistry.com/myimage:latest
17.15. Common DevOps
17.15.1. Files are in container
container path to code and files dcontainerpath :: /var/www/html store code and files to this folder :: html
17.15.1.1. Makefile at remote Prod/Live host /root/Makefile
- Prepare in container a script to dump db
/root/dump.sh
#!/bin/bash mysqldump -u root --password='$mypass$' livedbname > /root/myproject.sql
.PHONY: info rmbkfolder bkfolder bkdb bkdbsamename tarall tarallexcludesamename drytarallexcludesamename tarcode taruploads taruploadsexclude filename := $(shell date +%F) # WordPress runs in container dcontainer = mycontainername # website root folder dcontainerpath = /var/www/html # container db dump script dconainterdbdump = /root/dump.sh # db dump location in container dcontainerdbdumppath = /root # host backup folder /root/html bkpath = html tarexclude = --exclude='wp-content/uploads' --exclude='sql-admin' taruploadsexclude = --exclude='backupbuddy_backups' tarallexclude = --exclude='wp-content/uploads/backupbuddy_bakcups' --exclude='sql-admin' # myproject.sql projectname = myproject # prepare files to push to Live environment transferpath = /web-root/transfer-files/dev-to-prod/source/ info: @echo first run bkfolder @echo then run make tarcode and make taruploads @echo run make tarall @echo run make taruploadsexclude if some directories need to be excluded @echo Finally, run rmbkfolder to release space bkfolder: rmbkfolder @echo should use docker cp -a if it is supported docker cp -L $(dcontainer):$(dcontainerpath) $(bkpath) rmbkfolder: rm -rf $(bkpath) bkdb: docker exec $(dcontainer) $dcontainerdbdump docker cp $(dcontainer):$(dcontainerdbdumppath)/$(projectname).sql $(projectname)-$(filename).sql bkdbsamename: docker exec $(dcontainer) $dcontainerdbdump docker cp $(dcontainer):$(dcontainerdbdumppath)/$(projectname).sql $(projectname).sql tarall: tar -czpf all-$(filename).tar.gz -C $(bkpath) . tarallexcludesamename: tar -czpf all.tar.gz -C $(bkpath) . $(tarallexclude)) drytarallexcludesamename: tar -cvzpf - -C $(bkpath) . $(tarallexclude) tarcode: tar -czpf code-$(filename).tar.gz -C $(bkpath) . $(tarexclude) taruploads: tar -czpf uploads-$(filename).tar.gz -C $(bkpath)/wp-content/uploads . taruploadsexclude: tar -czpf uploads-exclude-$(filename).tar.gz -C $(bkpath)/wp-content/uploads . $(taruploadsexclude) # copy code.tar.gz to /myproject/code.tar.gz # cd /myproject && tar -xvzf code.tar.gz # make wp-content/uploads directory # mkdir wp-content/uploads && tar -xvzf uploads.tar.gz -C wp-content/uploads cleantars: rm -rf all.tar.gz # copy dev code to a transfer folder, remove wp-config.php and .htaccess pushcodetolive: rsync -av ./$(bkpath)/ $(transferpath) rm $(transferpath)wp-config.php rm $(transferpath).htaccess
Also prepare on Live host /root/devops-db.sh
#!/bin/bash cd /root && make bkdbsamename
And /root/devops-files.sh
#!/bin/bash cd /root && make bkfolder && make tarallexcludesamename
17.15.1.2. Sql dump inside container
Inside container /var/transfer/bash/dbdump.sh
#!/bin/bash # Move to MySQL Dump Folder : cd /var/transfer/sql-dump/ # Remove dump folder (if Exists): rm -rf dev-to-prod # Create and Enter Dump Folder : mkdir dev-to-prod cd dev-to-prod # MySQL Dump A Copy of the Database to the server : mysqldump -u root --password='dbpassword' dbname > containername.sql # Find && Replace Dev URL with Prod URL : sed 's|http://www\.dev.mywebsite.com\.com|http://www\.mywebsite\.com|g' containername.sql
17.15.1.3. local dev /root/Makefile
Create /root/myfile to store ssh plain password
echo '$mysshpass$' > myfile
Live environment has IP 1.2.3.4 Dev uses container devcontainer and database name is devdb Dev container map files to dev host at /myproject/html
make importlivedb
/root/Makefile
.PHONY: importlivedb importlivefiles liveip=1.2.3.4 liveuser=root livescriptdb=/root/devops-db.sh livescriptfiles=/root/devops-files.sh livedbfile=/root/myproject.sql livedbfilename=myproject.sql livetarfile=/root/all.tar.gz containername=devcontainer password=$$mydevdbpassEscapeDollarSigns$$ dbname=devdb htmlpath=/myproject/html htmlparent=/myproject apacheuserandgroup=www-data:www-data importlivedb: @echo generate sql dump on Live sshpass -f myfile ssh -o StrictHostKeyChecking=no $(liveuser)@$(liveip) '$(livescriptdb)' @echo download dump to here sshpass -f myfile scp -o StrictHostKeyChecking=no $(liveuser)@$(liveip):$(livedbfile) . # ssh host1 'mysqldump -u user --password='"'"'$(dbpw)'"'"' dbname > ~/cs-devops/db.sql' # scp host1:~/cs-devops/db.sql db/ @echo change url in db file before import sed -i 's|http://www\.myweb\.com|http://dev\.myweb\.com|g' $(livedbfilename) sed -i 's|http://myweb\.com|http://dev\.myweb\.com|g' $(livedbfilename) cat $(livedbfilename) | docker exec -i $(containername) /usr/bin/mysql -u root --password='$(password)' $(dbname) importlivefiles: sshpass -f myfile ssh -o StrictHostKeyChecking=no $(liveuser)@$(liveip) '$(livescriptfiles)' sshpass -f myfile scp -o StrictHostKeyChecking=no $(liveuser)@$(liveip):$(livetarfile) ./all.tar.gz rm -rf html mkdir html tar -xzf all.tar.gz -C ./html cp -a $(htmlpath)/wp-config.php . cp -a $(htmlpath)/.htaccess . rm -rf $(htmlpath) cp -a html $(htmlparent)/ cp -a wp-config.php $(htmlpath)/ cp -a .htaccess $(htmlpath)/ chown -R $(apacheuserandgroup) $(htmlpath) cleanupimport: rm -rf all.tar.gz html $(livedbfilename)
17.15.2. Check container status and restart services
#!/bin/bash container=mycontainer containerstatus=$(docker inspect -f {{.State.Running}} $container) if [ "$containerstatus" != "true" ]; then echo "$container container is not started. Restarting.." docker restart $container fi containerstatus=$(docker inspect -f {{.State.Running}} $container) if [ "$containerstatus" != "true" ]; then echo "By now container should be started but it is not.. Abort restarting mysql and apache in container" else echo "Check mysql status in container" docker exec $container service mysql status if [ $? = "0" ]; then echo "mysql is running" else echo "Restarting mysql in container.." docker exec $container service mysql restart docker exec $container service mysql status # command exit status, command exit code, 0 is successful if [ $? = "0" ]; then echo "mysql is restarted" else echo "mysql could not be restarted." fi fi echo "Check apache in container" docker exec $container service apache2 status if [ $? = "0" ]; then echo "apache is running" else echo "Restarting apache in container.." docker exec $container service apache2 restart docker exec $container service apache2 status if [ $? = "0" ]; then echo "apache is restarted" else echo "apache could not be restarted." fi fi fi
17.15.3. Replace string in .sql
https://github.com/tazotodua/useful-php-scripts/blob/master/database-modifier-when-migrating-wordpress https://interconnectit.com/products/search-and-replace-for-wordpress-databases/ https://en-ca.wordpress.org/plugins/wp-migrate-db/ https://wordpress.org/plugins/better-search-replace/
UPDATE wp_options SET option_value = replace(option_value, 'http://olddomain.com', 'http://newdomain.com') WHERE option_name = 'home' OR option_name = 'siteurl'; UPDATE wp_posts SET guid = replace(guid, 'http://olddomain.com','http://newdomain.com'); UPDATE wp_posts SET post_content = replace(post_content, 'http://olddomain.com', 'http://newdomain.com'); UPDATE wp_postmeta SET meta_value = replace(meta_value, 'http://olddomain.com', 'http://newdomain.com');
17.15.4. Download files
csftp=ftp://example.com/var/www/html/
csftpuser=a
csftpp=b
dfolder1:
wget --user=$(csftpuser) --password='$(csftpp)' -cr -l inf -nH --cut-dirs=3 -P bk/page/ $(csftp)folder1
dfolder2:
wget --user=$(csftpuser) --password='$(csftpp)' -cr -l inf -nH --cut-dirs=3 -P bk/page/ $(csftp)folder2
dall: dfolder1 dfolder2
17.16. Kubernetes - K8s
- Container
- a collection of software processes unified by one namespace, with ac cess to an OS kernal that it shares with other containers and little to no access between containers
- Not tied to infrastructure - only needs Docker Engine
- Run as isolated processes in user space on the host OS
- (no term)
- Virtual Machine
- one or many applications
- Necessary binaries and libraries
- The entire guest OS to interact with the applications
- Docker instance
- a runtime instance of a Docker image contains:
- A Docker image
- An execution environment
- A standard set of instructions
- Docker Engine
- runtime and packaging tools that must be installed on the hosts that run Docker
- Docker Store or Docker Hub
- online cloud service where users can store and share Docker images
- Kubernetes
- a platform to schedule and run containers on clusters of virtual machines. It runs on bare metal, virutal machines, private datacenter and public cloud
- Written in Go by Google
Master Node
- Kube API Server (communication)
- Scheduler (scheduling)
- Monitor created Pods which do not have a Node design yet and design the Pod to run on a specific Node
- Cluster Manager (controllers)
- Background threads that run tasks in a cluster. Carry several roles but compiled into 1 single binary
- worker state
- maintain the correct number of Pods. Now it's replaced by deployment and ReplicaSets
- ReplicaSet
- ensure that a specified number of replicas for a pod are running at all times
- deployment
- join services and Pods together
- handle access management
- etcd database
- simple distributed key value pairs which is all cluster data
- kubectl app
- job scheduling info, Pod details, stage info. CLI
- server info and authentication info to access Kube API Server
communicate back with Master Node
- kubelet process
- this agent sees if Pods have been designed to the Nodes, execute Pod containers via the container engin, mount and run Pod volume and secrets, is aware of Pod of Node states
- kube-proxy process
- Network Proxy and load balancer for the service, on a single Worker Node. It handles the networking routing for TCP and UDP Packets, performs connection forwarding. It might expose to the public internet
- (no term)
- Docker daemon
- Pods
- a Pod is a group of containers, the smallest unit that can be scheduled as a deployment in K8s
- shared storage, Linux name space, IP addresses amongst other things
- pending, running, succeeded, failed (any non-zero exit), CrashLoopBackOff (fail to start
- containers
- (no term)
- Docker Swarm
- Written in Go
17.17. Images
17.17.1. image:alpine
apk add --no-cache libstdc++ \ && apk add --no-cache --virtual .build-deps \ curl \ && apk del .build-deps # --virtual .somename means the packages to be installed (curl) and their dependencies are not installed to global packages. Once the virtual package .somename is apk del, the packages specified those dependencies are removed # Shell access, enter into the container docker exec -it my_container ash
17.17.2. image:debian
17.17.3. image:buildpack-deps:jessie-scm
Parent: buildpack-deps:jessie-curl Available: git, curl, wget, openssh-client
17.17.4. image:buildpack-deps:stretch-curl
Parent: 2
17.17.5. image:php:5.6.31-apache-jessie image:php:apache
Short name: php:5.6.31-apache Parent: 1 Dockerfile
Refer to apache:config
- PHP_INI_DIR
- image:php:/usr/local/etc/php (/conf.d)
APACHE_CONFDIR :: etc/apache2 APACHE_ENVVAR :: /etc/apache2/envvars APACHE_RUN_USER :: www-data APACHE_DOCUMENT_ROOT, WORKDIR :: /var/www/html APACHE_LOG_DIR :: /var/log/apache2
EXPOSE 80 CMD ["apache2-foreground"] /usr/local/bin/apache2-foreground
/var/log/apache2/error.log => /dev/stderr /var/log/apache2/access.log => /dev/stdout /var/log/apache2/other_vhosts_access.log => /dev/stdout
copy host config/php.ini to usr/local/etc/php/php.ini (name has to be exact) or copy php-prod.ini to /usr/local/etc/php/conf.d
docker-php-ext-install uses docker-php-ext-enable which adds docker-php-ext-{ext-name}.ini to usr/local/etc/php/conf.d
If you want to get some from PHP's source, you can extract, use and delete it.
FROM php:7.0-apache RUN docker-php-source extract \ # do important things \ && docker-php-source delete
Install PHP Core Extensions
FROM php:7.0-fpm RUN apt-get update && apt-get install -y \ libfreetype6-dev \ libjpeg62-turbo-dev \ libmcrypt-dev \ libpng-dev \ && docker-php-ext-install -j$(nproc) iconv mcrypt \ && docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \ && docker-php-ext-install -j$(nproc) gd
Install PECL Extensions use docker-php-ext-enable
FROM php:7.1-fpm RUN pecl install redis-3.1.0 \ && pecl install xdebug-2.5.0 \ && docker-php-ext-enable redis xdebug FROM php:5.6-fpm RUN apt-get update && apt-get install -y libmemcached-dev zlib1g-dev \ && pecl install memcached-2.2.0 \ && docker-php-ext-enable memcached
Install other extensions
FROM php:5.6-apache RUN curl -fsSL 'https://xcache.lighttpd.net/pub/Releases/3.2.0/xcache-3.2.0.tar.gz' -o xcache.tar.gz \ && mkdir -p xcache \ && tar -xf xcache.tar.gz -C xcache --strip-components=1 \ && rm xcache.tar.gz \ && ( \ cd xcache \ && phpize \ && ./configure --enable-xcache \ && make -j$(nproc) \ && make install \ ) \ && rm -r xcache \ && docker-php-ext-enable xcache
The docker-php-ext-* scripts can accept an arbitrary path, but it must be absolute (to disambiguate from built-in extension names), so the above example could also be written as the following:
FROM php:5.6-apache RUN curl -fsSL 'https://xcache.lighttpd.net/pub/Releases/3.2.0/xcache-3.2.0.tar.gz' -o xcache.tar.gz \ && mkdir -p /tmp/xcache \ && tar -xf xcache.tar.gz -C /tmp/xcache --strip-Components=1 \ && rm xcache.tar.gz \ && docker-php-ext-configure /tmp/xcache --enable-xcache \ && docker-php-ext-install /tmp/xcache \ && rm -r /tmp/xcache
copy host config/php.ini to usr/local/etc/php/php.ini (name has to be exact) or copy php-prod.ini to /usr/local/etc/php/conf.d
17.17.5.1. Sample
FROM php:5.6.31-apache-jessie # apt-utils has to be installed to avoid throwing errors during installation on Debian RUN set -ex; \ \ apt-get update && apt-get install -y \ apt-utils \ nano \ libfreetype6-dev \ libmcrypt-dev \ libjpeg62-turbo-dev \ libpng-dev \ && \ apt-get clean && rm -rf /var/lib/apt/lists/* && \ \ docker-php-ext-install -j$(nproc) iconv mcrypt && \ docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ && \ docker-php-ext-install -j$(nproc) gd mysqli mysql pdo pdo_mysql pcntl zip opcache COPY config/php-prod.ini /usr/local/etc/php/conf.d/ RUN { \ echo 'opcache.memory_consumption=128'; \ echo 'opcache.interned_strings_buffer=8'; \ echo 'opcache.max_accelerated_files=4000'; \ echo 'opcache.revalidate_freq=2'; \ echo 'opcache.fast_shutdown=1'; \ echo 'opcache.enable_cli=1'; \ } > /usr/local/etc/php/conf.d/opcache-recommended.ini # enable apache modules RUN a2enmod rewrite expires VOLUME /var/www/html # TODO chown -R www-data:www-data /var/www/html CMD ["apache2-foreground"]
17.17.6. image:php53
17.17.6.1. image:bylexus:apache-php53
Ubuntu 12.04 (LTS), Apache 2.2, PHP 5.3.10 Ubuntu Server 12.04, based on ubuntu docker image apache2 php5 php5-cli libapache2-mod-php5 php5-gd php5-ldap php5-mysql php5-pgsql
docker pull bylexus/apache-php53 https://hub.docker.com/r/bylexus/apache-php53/
document root /var/www
docker run -d -p 8080:80 \ -v /home/user/webroot:/var/www \ -e PHP_ERROR_REPORTING='E_ALL & ~E_STRICT' \ bylexus/apache-php53 -v [local path]:/var/www maps the container's webroot to a local path -p [local port]:80 maps a local port to the container's HTTP port 80 -e PHP_ERROR_REPORTING=[php error_reporting settings] sets the value of error_reporting in the php.ini files.
Apache Logs docker logs -f container-id
- ErrorLog /dev/stdout
- CustomLog /dev/stdout combined
/etc/php5/apache2/php.ini /etc/php5/cli/php.ini
Default:
- display_errors=ON
- error_reporting=E_ALL & ~E_DEPRECATED & ~E_NOTICE
- change it via -e PHP_ERROR_REPORTING=…
17.17.6.2. image:orsolin/php:5.3-apache
https://github.com/cristianorsolin/docker-php-5.3-apache
This one is Debian and has curl
phphoht:
image: orsolin/docker-php-5.3-apache
container_name: phphoht
volumes:
- "./:/var/www/html"
networks:
app_nethoht:
ipv4_address: 172.19.2.3
ports:
- "32003:80"
depends_on:
- dbhoht
links:
- dbhoht
17.17.7. image:php:7
- Tag latest is 7.2.3-cli-stretch
- php:7.2.3-apache image:php:7:apache
- ftp mysqlnd mbstring ftp password-argon2 sodium curl libedit openssl zlib
17.17.9. image:python
python:3python:3.8.2-buster
Basics
# Run Python Interpreter interactively docker run -it --rm --name mypy python:3 #Map current folder and run hello.py docker run -it --rm --name mypy -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:3.8.2-buster python hello.py
17.17.11. image:mysql
- https://hub.docker.com/_/mysql
- Parent
- 5.7.9
- 1
- 5.7.40
- image:oraclelinux:7-slim
- Before MySQL v8
- no ARM version
- mysql:mysql
- mysqld
EXPOSE 3306-v /my/mysql-data:/var/lib/mysql-v /my/mysql-log:/var/log/mysql-v /my/mysql-conf:/etc/mysql/conf.d- Main my.cnf can be overriden
-v /my/my.cnf:/etc/mysql/my.cnf
- if you don't want to throw in my.cnf or conf.d
docker exec -it some-mysql bash- ARM64
- Start with arm64v8/mysql:8.0
17.17.11.1. Load configs
- Startup config
/etc/mysql/my.cnfand any.cnffiles under/etc/mysql/conf.d- (no term)
- Mount host directory to
/etc/mysql/conf.din container using-vin docker run - (no term)
- Instead of loading custom
conf.ddirectory, tags--tag-namecan be used adjust the configs - (no term)
- Use 3.1 or docker-compose command
- (no term)
For container that couldn't have more than 128M memory, use
innodb-buffer-pool-sizeto scale it downdocker run -it --rm --name my-mysql-container \ -e MYSQL_ROOT_PASSWORD=my-secret-pw \ -d mysql:5.7.9 \ --character-set-server=utf8mb4 \ --collation-server=utf8mb4_unicode_ci --innodb-buffer-pool-size=50M
In docker-compose.yml
command: mysqld --innodb-buffer-pool-size=50M- (no term)
- More MySQL Config
17.17.11.2. Environment vars
- None of these will have effect if the container is started with a data dir that already contains a db:
- MYSQL_ROOT_PASSWORD
- required
- MYSQL_DATABASE
- opt. a db to be created on image startup
- MYSQL_USER, MYSQL_PASSWORD
- opt. create a user for MYSQL_DATABASE
- (no term)
These vars can be loaded through a file
docker run -it --rm -e MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql-root mysql:5.7
- e.g.
TZ=America/Toronto
17.17.11.3. Import .sql on image startup
Poibt a directory that has a .sh, .sql, and .sql.gz to a dir in container
-v /host/path/to/db-dir:/docker-entrypoint-initdb.d
Importing needs a lot of memory. If host doesn't have 128M memory, try to import it locally and then rsync the persist data folder to host and chown -R 999:docker /path/to/persistdir accordingly on host.
17.17.11.4. Access, view logs, db dump
docker exec -it my-mysql-container bash docker logs my-mysql-container docker exec some-mysql sh -c \ 'exec mysqldump --all-databases -uroot -p"$MYSQL_ROOT_PASSWORD"' > /some/path/on/your/host/all-databases.sql
17.17.11.5. Persist data
- See 17.8.2
A place to persist db data on host :: /my/own/datadir
-v /my/own/datadir:/var/lib/mysql
For dev where db data can be imported and disposed when needed, use volume (e.g. dbdata)
version: '2'
volumes:
dbdata:
networks:
app_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.18.0.0/16
gateway: 172.18.0.1
services:
db:
image: mysql:5.7.9
container_name: db
volumes:
- "dbdata:/var/lib/mysql"
- "/my/own/datadir:/docker-entrypoint-initdb.d"
networks:
app_net:
ipv4_address: 172.18.0.2
restart: always
environment:
MYSQL_ROOT_PASSWORD: d7
MYSQL_DATABASE: d7
MYSQL_USER: d7
MYSQL_PASSWORD: d7
17.17.11.6. Example
To initialize a db, use docker run with data persist
docker run -it --name my-db-container \ -v /host/path/to/dbdata:/var/lib/mysql \ -v /host/path/to/db-dir:/docker-entrypoint-initdb.d \ -e MYSQL_ROOT_PASSWORD=d7 \ -e MYSQL_DATABASE=d7 \ -e MYSQL_USER=d7 \ -e MYSQL_PASSWORD=d7 \ -d mysql:5.7.9 \ --innodb-buffer-pool-size=50M \ --innodb-buffer-pool-chunk-size=50M \ --max-allowed-packet=50M
Check db status, then you docker-compose restart the container without specifying import directive in docker-compose.yml
mydbcontainer:
image: mysql:5.7.9
container_name: mydbcontainer
volumes:
- "/host/path/to/dbdata:/var/lib/mysql"
command:
mysqld --innodb-buffer-pool-size=50M --innodb-buffer-pool-chunk=50M --max-allowed-packet=16M
restart: always
networks:
...
version: '3.5' services: mydb: container_name: mydb image: mysql:5.7.40 platform: linux/amd64 command: --default-authentication-plugin=mysql_native_password volumes: # Include this line for importing .sql to initialize database mymysqldb # - ../../../my-mysql-data:/var/lib/mysql:cached - ./setupfiles/mysql/my.cnf:/etc/mysql/my.cnf - mydbdata: /var/lib/mysql environment: - MYSQL_ROOT_PASSWORD=myrootpass # Include this line for imporiting .sql to initialize database mymysqldb - MYSQL_DATABASE=mymysqldb ports: - "9006:3306" networks: - default
17.17.12. image:wordpress image:wordpress:apache
- wordpress:4.9.3-php7.1-apache
- Parent
- php:7.1-apache image:php:apache image:php:7:apache
- ghostscript libfreetype6-dev libjpeg-dev libmagickwand-dev libpng-dev libzip-dev
- php:gd mysqli opcache bcmath exif zip
17.17.12.1. docker-compose.yml
version: '2' volumes: dbsandboxwp: networks: app_netsandboxwp: driver: bridge ipam: driver: default config: - subnet: 172.20.0.0/24 gateway: 172.20.0.1 services: dbsandboxwp: image: mysql:5.7.9 container_name: dbsandboxwp networks: app_netsandboxwp: ipv4_address: 172.20.0.2 ports: - "31000:3306" restart: always environment: MYSQL_ROOT_PASSWORD: wp MYSQL_DATABASE: wp MYSQL_USER: wp MYSQL_PASSWORD: wp phpsandboxwp: image: wordpress:4.9.3-php7.1-apache container_name: phpsandboxwp volumes: - "./:/var/www/html" restart: always networks: app_netsandboxwp: ipv4_address: 172.20.0.3 environment: WORDPRESS_DB_HOST: dbsandboxwp WORDPRESS_DB_PASSWORD: wp WORDPRESS_DB_USER: wp WORDPRESS_DB_NAME: wp WORDPRESS_TABLE_PREFIX: wp_ # environment variables honor mysql # WORDPRESS_DB_HOST WORDPRESS_DB_PASSWORD # environment variables default # WORDPRESS_DB_USER root # WORDPRESS_DB_NAME wordpress # WORDPRESS_TABLE_PREFIX "" ports: - "32000:80" depends_on: - dbsandboxwp links: - dbsandboxwp
17.17.13. image:drupal image:drupal:apache
drupal:7.56-apache Parent :: php:7.0-apache
17.17.13.1. docker-compose.yml
docker run -ti –rm –volumes-from phpsandboxd7 –name html-data ubuntu:xenial docker cp html-data:/var/www/html .
version: '2' volumes: volsandboxd7: networks: app_netsandboxd7: driver: bridge ipam: driver: default config: - subnet: 172.20.1.0/24 gateway: 172.20.1.1 services: dbsandboxd7: image: mysql:5.7.9 container_name: dbsandboxd7 networks: app_netsandboxd7: ipv4_address: 172.20.1.2 aliases: - mysql ports: - "31001:3306" restart: always environment: MYSQL_ROOT_PASSWORD: d7 MYSQL_DATABASE: d7 MYSQL_USER: d7 MYSQL_PASSWORD: d7 phpsandboxd7: image: drupal:7.56-apache container_name: phpsandboxd7 volumes: - volsandboxd7:/var/www/html # - /var/www/html/modules # - /var/www/html/profiles # - /var/www/html/sites # - /var/www/html/themes restart: always networks: app_netsandboxd7: ipv4_address: 172.20.1.3 aliases: - drupal ports: - "32001:80" depends_on: - dbsandboxd7
17.17.14. image:tatemz/wp-cli image:tatemz/wp-cli
https://github.com/tatemz/docker-wpcli
Refer to wp-cli:search-replace
docker-compose run --rm my-wpcli docker-compose run --rm my-wpcli db check docker-compose run --rm my-wpcli post list docker-compose run --rm my-wpcli db search 'https?://' --regex --stats # For some reason, the following only works when run in PowerShell # This replaces A with B for the current database without changing it but instead export it. docker-compose run --rm my-wpcli search-replace 'http://live.ca' 'http://localhost:9999' --export=/var/www/html/devops/db/export.sql # There's no way to search/replace multiple strings. Do it one by one docker-compose run --rm my-wpcli search-replace 'https://live.ca' 'http://localhost:9999' # at the end export it docker-compose run --rm my-wpcli db export --add-drop-table /var/www/html/devops/db/dump.sql
17.17.15. image:wordpress:cli image:wordpress:cli
wordpress:cli-php5.6 wordpress-cli-2-php5.6 wordpress:cli-2.0-php5.6 wordpress:cli-2.0.0-php5.6 wordpress:cli-php7.0 wordpress:cli-php7.1 wordpress:cli-php7.2
docker pull wordpress:cli-php7.0 docker run -it --rm --volumes-from mywp --network container:mywp wordpress:cli-php7.0 user list
17.17.16. image:lucee docker:image:lucee
- Image name
- lucee/lucee4-nginx
- Tags
- https://hub.docker.com/r/lucee/lucee4-nginx/tags
- Choose a tag
- https://github.com/lucee/lucee-dockerfiles
- Parent Images
- lucee/lucee4 > tomcat:8.0.36-jre8 > openjdk:8-jre > buildpack-deps:stretch-curl
17.17.16.1. docker-compose.yml Dockerfile
docker-compose.yml
version: '2' networks: app_net: driver: bridge ipam: driver: default config: - subnet: 172.16.2.0/24 gateway: 172.16.2.1 services: lucee: build: . image: cflucee:latest container_name: lucee ports: - "80:80" - "8888:8888" volumes: - /home/you/db/access:/root/db/ - /home/you/www:/var/www # Scheduled tasks are setup in http://127.0.0.1:8888/lucee/admin/web.cfm # /home/you/www/WEB-INF/lucee/scheduler/scheduler.xml > /var/wwww/WEB-INF/lucee/scheduler/scheduler.xml - /home/you/logs/tomcat:/usr/local/tomcat/logs - /home/you/logs/nginx:/var/log/nginx - /home/you/logs/lucee:/opt/lucee/web/logs networks: app_net: ipv4_address: 172.16.2.3
Dockerfile
- Change it based on the base Dockerfile
- https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/Dockerfile
- Image has 2 ports exposed
- 8080 (Tomcat listens but not used) 8888 (used for Lucee Server Admin/Web)
- (no term)
- Config folders
- Tomcat
/usr/local/tomcat/conf- Lucee config for default site
/opt/lucee/web- Lucee server context
/opt/lucee/server/lucee-server/context- Lucee web
/opt/lucee/web
- (no term)
- Log folders
- Tomcat
/usr/local/tomcat/logs- Lucee logs for default site
/opt/lucee/web/logs
- (no term)
- Environment varialbes
FROM lucee/lucee4-nginx:4.5.1.024 ENV TZ=America/Toronto RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone # Copy UcanAccess COPY ucanaccess/* /usr/local/tomcat/lib/ # Custom tags are manually copied to each host's root folder... COPY config/customtags/* /opt/lucee/web/customtags/ # COPY config/customtags/* /opt/lucee/server/lucee-server/context/library/ # Tomcat configs # COPY catalina.properties server.xml web.xml /usr/local/tomcat/conf/ COPY config/tomcat/server.xml /usr/local/tomcat/conf/ COPY config/tomcat/context.xml /usr/local/tomcat/conf/ # COPY config/tomcat/legacy_js.xml /usr/local/tomcat/conf/Catalina/127.0.0.1/legacy_js.xml # Custom setenv.sh to load Lucee COPY setenv.sh /usr/local/tomcat/bin/ RUN chmod a+x /usr/local/tomcat/bin/setenv.sh # Default setenv.sh https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/setenv.sh # Added UcanAccess # NGINX configs COPY config/nginx/ /etc/nginx/ # /etc/nginx/nginx.conf :: Default not modified https://github.com/lucee/lucee-dockerfiles/blob/master/lucee-nginx/4.5/nginx.conf # /etc nginx/default.conf :: https://github.com/lucee/lucee-dockerfiles/blob/master/lucee-nginx/4.5/default.conf # Add datasources. Default https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/lucee-server.xml COPY config/lucee/lucee-server.xml /opt/lucee/server/lucee-server/context/lucee-server.xml # Lucee server PRODUCTION configs # COPY config/lucee/lucee-web.xml.cfm /opt/lucee/web/lucee-web.xml.cfm # Deploy codebase to container # COPY index.cfm /var/www
17.17.16.2. Add ucanaccess db driver to TomCat, Add M$ Access Datasources
- Refer to docker:image:tomcat
- Add Access datasource on
yourdomain.com:8888/lucee/admin/server.cfm - Other - JDBC Driver
- abc
- net.ucanaccess.jdbc.UcanaccessDriver
jdbc:ucanaccess:///home/MyName/www/database/MyMsAccessDB.mdb
Create a datasource using UI. One of the Lucee setting files will have this line in <datasources>
<data-source allow="511" blob="false" class="net.ucanaccess.jdbc.UcanaccessDriver" clob="false" connectionTimeout="1" custom="" database="" dbdriver="Other" dsn="jdbc:ucanaccess:///home/MyName/www/database/MyMsAccessDB.mdb" host="" metaCacheTimeout="60000" name="abc" password="abc test password" storage="false" validate="false"/>
17.17.16.3. Lucee server and web settings
- Lucee server setting
/opt/lucee/server/lucee-server/context/lucee-server.xml- Individual website setting
/opt/lucee/web/lucee-web.xml.cfm
17.17.16.4. Custom tags (.cfm|.cfc) under /opt/lucee/web/customtags/
17.17.16.5. Change Lucee Server Admin UI password
Use this with a salt to generate hspw in Lucee server setting <cfLuceeConfiguration hspw="" salt="" version="4.5">
Refer to image:lucee:salt. It seems like you should use the salt that is in the default lucee-server.xml.
Can't use a random salt.
17.17.16.6. How does it work with Nginx?
Lucee operates on TomCat port 8888 using 127.0.0.1. Nginx forwards 80 incoming traffic to port 8888.
By default, only *.cfm|cfc|cfr files are forwarded to 8888. Add more files to /etc/nginx/conf.d/default.conf
location ~* \.(cfm|cfc|cfr|jpe?g|js|css|png|gif|html)$ { proxy_pass http://127.0.0.1:8888; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; }
17.17.16.7. Virtual Directory and TomCat
Virtual Directory. Setup docker:image:tomcat:vd first, then setup forwarding
location ~* /legacy_js { proxy_pass http://127.0.0.1:8888; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; }
17.17.16.8. Add hosts
server { listen 80 default_server; server_name domain1.com; index index.cfm index.html index.htm; root /var/www/site1; server_name_in_redirect off; include allcf/common.conf; location ~* (/legacy_js)|(\.(cfm|cfc|cfr|jpe?g|js|gif|css|png|html)$) { proxy_pass http://127.0.0.2:8888; proxy_redirect off; proxy_set_header Host 127.0.0.2; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; } } server { listen 80; server_name domain2.com; index index.cfm index.html index.htm; root /var/www/site2; server_name_in_redirect off; include allcf/common.conf; location ~* (/legacy_js)|(\.(cfm|cfc|cfr|jpe?g|js|gif|css|png|html)$) { proxy_pass http://127.0.0.3:8888; proxy_redirect off; proxy_set_header Host 127.0.0.3; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; } }
Add hosts in docker:image:tomcat:host
17.17.16.9. Scheduled Tasks
Create them in /lucee/admin/web.cfm
Checkout this file /var/www/WEB-INF/lucee/scheduler/scheduler.xml
17.17.17. image:tomcat docker:image:tomcat
Image name: tomcat:8.0.36-jre8
17.17.17.1. Settings
The default webapps (including TomCat Admin Console, localhost:8080/manager/html and .jar files) is in /usr/local/tomcat/webapps The default connector port is 8080.
docker:image:lucee deletes this webapps folder, place it under /usr/local/tomcat/lucee which contains all .jar files.
Include that directory in catelina.properties in common.loader=${catalina.home}/lucee/*.jar
Define a <web-app> with multiple servlets in ${catalina.home}/web.xml to handle all .html and .cf(m|c) files.
When the web-app and Lucee servlet are built, new Lucee admin paths (context) are built :: 127.0.0.1:8888/lucee/admin/server.cfm and web.cfm Corresponding paths are /opt/lucee/server(/lucee-server) and opt/lucee/web
In ${catalina.home}/server.xml, define Host(s) as usual. Each <Host> has appBase="webapps" which is a folder under /usr/local/tomcat. Using the empty webapps folder is fine.
Each <Host> can have multiple <Context> with different path/docBase. The root directory of each Host can have:
- META-INF folder
- context.xml (server-dependent config, e.g. datasource),
- WEB-INF folder
- web.xml (server-independent config, e.g. servlet mappings, docker:image:lucee:tasks)
If there's ${catalina.home}/context.xml, all hosts have these settings and
override <Context> set in <Host> in server.xml and override <Context> in any META-INF/context.xml under all hosts:
${catalina_home}/[enginename]/[host]/META-INF/context.xml
The only time path attribute in <Context> has any effect is in the server.xml
17.17.17.2. Install UcanAccess JDBC driver to enable M$ Access datasource
Copy these jar files
COPY ucanaccess/* /usr/local/tomcat/lib/
ucanaccess-4.0.1.jar lib/*.jar (commons-lang, commons-logging, hsqldb, jackcess)
Add these 2 lines to setenv.sh
UCANACCESS_HOME=/usr/local/tomcat/lib; export UCANACCESS_HOME;
COPY setenv.sh /usr/local/tomcat/bin/ RUN chmod a+x /usr/local/tomcat/bin/setenv.sh
17.17.17.3. Setup Virtual Directory per Host docker:image:tomcat:vd
In /usr/local/tomcat/conf/server.xml, you can see how many hosts are defined. e.g.
<Server port="8005" shutdown="SHUTDOWN"> <Listener className="org.apache.catalina.startup.VersionLoggerListener" /> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> <Listener className="..." /> <Service name="Catalina"> <Connector executor="tomcatThreadPool" port="8888" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" /> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> <Engine name="Catalina" defaultHost="127.0.0.1"> <Host name="127.0.0.1" appBase="webapps" unpackWARs="true" autoDeploy="true"> <Valve className="org.apache.catalina.valves.RemoteIpValve" remoteIpHeader="X-Forwarded-For" requestAttributesEnabled="true" /> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t "%r" %s %b" /> <Context path="" docBase="/var/www/"> <JarScanner scanClassPath="false"/> </Context> <!-- You can define virtual directory here or follow the way of adding vd.xml--> <!-- <Context path="/legacy_js" docBase="/path/outside/var/www/legacy_js"></Context> --> </Host> </Engine> </Service> </Server>
Method 1 is to add a <Context> tag in a <Host> in server.xml. But for me it doesn't work when multiple hosts need to have the same vd and it creates a lot of other problems.
General rule :: don't include more than 1 <Context> in each <Host> in server.xml
Method 2 is to create a file legacy_js.xml under ${catalina_home}/conf/Catalina/127.0.0.1/ Where Catalina is the engine name and 127.0.0.1 is the Host name. #+NAME legacy_js.xml
<?xml version='1.0' encoding='utf-8'?> <Context docBase="/root/vd/legacy_js/"></Context>
http://127.0.0.1/legacy_js/abc.js points to /root/vd/legacy_js/abc.js
If it's foo#bar.xml then http://127.0.0.1/foo/bar/abc.js
Method 2 is not tested for multiple hosts usage…
Method 3 is tested.
Create relative symbolic link for docker usage. /var/www/site2/legacy_js point to ../site1/source_legacy_js
cd /var/www/site2 ln -s ../site1/source_legacy_js/ legacy_js
TomCat can't follow relative symbolic link.. Add this line to ${catalina_home}/context.xml for TomCat 8
<Resources allowLinking="true"></Resources> ;; # For TomCat 7, add attribute allowLinking="true" to the only <Context>
I recently found out Method 3 doesn't work.. Here's Method 4. Don't reverse proxy to TomCat at all, if vd has only static files. In nginx default.conf
location ~* ^/legacy_js {
;; # nginx serve the static files
}
location ~* \.(cfm|cfc|cfr|jpe?g|js)
17.17.17.4. Add host docker:image:tomcat:host
Host name should be 127/8 ip address otherwise you need to setup DNS. Refer to docker:image:tomcat:log
<Host name="127.0.0.2" appBase="webapps">
<Valve className="org.apache.catalina.valves.RemoteIpValve"
remoteIpHeader="X-Forwarded-For"
requestAttributesEnabled="true" />
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
<Context path="" docBase="/var/www/site1/">
<JarScanner scanClassPath="false"/>
</Context>
</Host>
<Host name="127.0.0.3" appBase="webapps">
<Valve className="org.apache.catalina.valves.RemoteIpValve"
remoteIpHeader="X-Forwarded-For"
requestAttributesEnabled="true" />
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
<Context path="" docBase="/var/www/site2/">
<JarScanner scanClassPath="false"/>
</Context>
<Context docBase="/var/www/vd/legacy_js/" path="/legacy_js"></Context>
</Host>
17.17.17.5. Restrict Access
Refer to docker:image:tomcat:log
<!-- RemoteAddrValue is %a in log -->
<Valve className="org.apache.catalina.valves.RemoteAddrValve"
allow="allow.com" />
<!-- RemoteHostValue is %h in log -->
<Valve className="org.apache.catalina.valves.RemoteHostValve"
allow="127.0.0.1,10.100.1.2" />
17.17.17.6. Logging
docker:image:tomcat:log Google :: tomcat access log value pattern
- %h
- remote host name. Or IP if enableLookups for the connector is false If request is passed from Nginx (proxy_pass), it's 127.0.0.1 If request is directly from Internet, it's the requester's IP
- %a
- Remote IP address. Same as %h but IP only If request is passed from Nginx (proxy_pass), it's 127.0.0.1 If request is directly from Internet, it's the requester's IP.
- %v
- Local server name. If request is passed from Nginx (proxy_pass), it's the name of <Host> in Tomcat If request is directly from Internet, it's the request domain name
- %A
- Local IP address. Tomcat server public IP
- %{local}p
- local port. Which is 80 or 8888 where is set in <Connector>
- %{remote}p
- remote port. IDK.. Maybe random
- %{xxx}i
- incoming header value. e.g. %{X-Forwarded-Host}
- %{xxx}o
- outgoing header value
17.17.18. image:goaccess docker:image:goaccess
- Don't follow the provided instructions..
- https://github.com/allinurl/goaccess#docker
- (no term)
Follow mine
git clone https://github.com/allinurl/goaccess.git goaccess && cd $_ docker build . -t allinurl/goaccess mkdir -p /home/li/goaccess/{data,html,log} # create config file nano /home/li/goaccess/data/goaccess.conf # restart container after config file is changed docker restart goaccess docker run --restart=always -d -p 7890:7890 \ -v "/home/li/goaccess/data:/srv/data" \ -v "/home/li/goaccess/html:/srv/report" \ -v "/home/li/goaccess/log:/srv/logs" \ --name=goaccess allinurl/goaccess \ --no-global-config --config-file=/srv/data/goaccess.conf \ --log-file=/srv/logs/access.log # entrypoint is `/bin/goaccess` # default conf file is in the Alpine container :: /etc/goaccess # so must use --config-file to override # use --log-file to dynamically point to the log you want to analyze # generated HTML report is under /srv/goaccess/html/index.html # open it locally on host machine will have a real-time updated one! Because the webpage uses WebSocket # completely remove docker stop goaccess docker rm goaccess docker rmi allinurl/goaccess
17.17.18.1. Config goaccess.conf
- https://pantheon.io/docs/nginx-access-log
- https://raw.githubusercontent.com/allinurl/goaccess/master/config/goaccess.conf
- https://goaccess.io/man#custom-log
- Use a script file to convert nginx:d:log_format to goaccess format. For example
./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'- Then copy the time-format, date-format and change log_format to log-format and copy/paste to the conf file
- Use a script file to convert nginx:d:log_format to goaccess format. For example
Better to add the following lines to the default conf or you can just have these lines in the conf
log-file /srv/logs/access.log output /srv/report/index.html real-time-html true # real-time-html is true is to ensure `docker run -d` runs continuously # followed by script converted log-format, time-format, etc. # log-format COMBINED
17.17.19. image:stilliard/pure-ftpd docker:image:ftp
Image name: stilliard/pure-ftpd:hardened Parent image: debian:jessie
https://github.com/stilliard/docker-pure-ftpd By default, it only supports FTP not SFTP
Dockerfile docker build -t myftp .
FROM stilliard/pure-ftpd:hardened # e.g. you could change the defult command run: CMD /run.sh -c 30 -C 5 -l puredb:/etc/pure-ftpd/pureftpd.pdb -E -j -R -P $PUBLICHOST -p 30000:30009 # /usr/sbin/pure-ftpd # -c 50, --maxclientsnumber (no more than 50 people at once) # -C 10, --maxclientsperip (no more than 10 requests from the same ip) # -l puredb:/etc/pure-ftpd/pureftpd.pdb, --login (login file for virtual users) # -E, --noanonymous (only real users) # -j, --createhomedir (auto create home directory if it doesn't already exist) # -R, --nochmod (prevent usage of the CHMOD command) # -tls 1, Enables optional TLS support # In Compose # -P $PUBLICHOST :: setting for PASV support, passed in your the PUBLICHOST env var # -p 30000:30009 :: PASV port range
docker-compose.yml
version: '2' networks: app_net: driver: bridge ipam: driver: default config: - subnet: 172.16.3.0/24 gateway: 172.16.3.1 services: myftp: build: . image: myftp:latest container_name: myftp environment: - PUBLICHOST=ftp.yourdomain.com # - ADDED_FLAGS=-d -d ports: - "21:21" - "30000-30009:30000-30009" # open these many ports to support Passive Mode.. volumes: - /host/storefiles/ftp:/home/ftpusers - /host/config/ftp/passwd:/etc/pure-ftpd/passwd # persist the user database # only one file under /etc/pure-ftpd/passwd which is pureftps.passwd and it contains the user database # - /host/config/ftp/ssl:/etc/ssl/private # a single `pure-ftpd.pem` file with server's SSL certificates for TLS support networks: app_net: ipv4_address: 172.16.3.3
Makefile
FTPCONFIG=/etc/pure-ftpd/passwd/pureftpd.passwd .PHONY: default info start stop restart list adduser deluser default: @echo "make [command]. Available commands are:" @echo "info, start, stop, restart, list, adduser, deluser" info: @echo "List all virutal user: make list" @echo "Add a new user abc and make parent folder to /FTP" @echo 'make adduser username="abc" folder="FTP"' @echo 'Add a new user abc and make parent folder to root /' @echo 'make adduser username="abc" folder="application1"' @echo 'Delete a user abc: make deluser username="abc"' start: sudo docker-compose up -d --build stop: sudo docker-compose down -v restart: sudo docker-compose down -v && sudo docker-compose up -d --build enter: sudo docker exec -it newcomftp bash -I list: @echo "<account>:<password>:<uid>:<gid>:<gecos>:<home directory>:<upload bandwidth>:<download bandwidth>:<upload ratio>:<download ratio>:<max number of connections>:<files quota>:<size quota>:<authorized local IPs>:<refused local IPs>:<authorized client IPs>:<refused client IPs>:<time restrictions>" sudo cat ~/config/ftp/passwd/pureftpd.passwd adduser: sudo docker exec -it myftp pure-pw useradd $$username -f $(FTPCONFIG) -m -u ftpuser -d /home/ftpusers/$$folder deluser: sudo docker exec -it myftp pure-pw userdel $$username -f $(FTPCONFIG) -m
Pureftp Manual
ftp:passive mode FTP opens one channel for command and another for data.
- Active mode
- client establishes the command channel (from client port X to server port 21) but the server establishes the data channel (from server port 20 to client port Y, where Y has been supplied by the client).
- Passive mode
- client establishes both channels. In that case, the server tells the client which port should be used for the data channel.
17.17.20. image:dockerfile-from-image
image:dockerfile-from-image https://github.com/levonlee/dockerfile-from-image
docker run -v /var/run/docker.sock:/var/run/docker.sock dockerfile-from-image <IMAGE_TAG_OR_ID> e.g. docker run -v /var/run/docker.sock:/var/run/docker.sock dockerfile-from-image ubuntu
For windows docker run -v //var/run/docker.sock:/var/run/docker.sock dockerfile-from-image ubuntu
Dockerfile (added one line: RUN chmod)
FROM alpine:3.2 MAINTAINER CenturyLink Labs <clt-labs-futuretech@centurylink.com> RUN apk --update add ruby-dev ca-certificates && \ gem install --no-rdoc --no-ri docker-api && \ apk del ruby-dev ca-certificates && \ apk add ruby ruby-json && \ rm /var/cache/apk/* ADD dockerfile-from-image.rb /usr/src/app/dockerfile-from-image.rb RUN chmod +x /usr/src/app/dockerfile-from-image.rb ENTRYPOINT ["/usr/src/app/dockerfile-from-image.rb"] CMD ["--help"]
FROM alpine:3.2
MAINTAINER CenturyLink Labs <clt-labs-futuretech@centurylink.com>
RUN apk --update add ruby-dev ca-certificates && \
gem install --no-rdoc --no-ri docker-api && \
apk del ruby-dev ca-certificates && \
apk add ruby ruby-json && \
rm /var/cache/apk/*
ADD dockerfile-from-image.rb /usr/src/app/dockerfile-from-image.rb
RUN chmod +x /usr/src/app/dockerfile-from-image.rb
ENTRYPOINT ["/usr/src/app/dockerfile-from-image.rb"]
CMD ["--help"]
dockerfile-from-image.rb (same as GitHub)
#! /usr/bin/env ruby
require 'docker'
require 'optparse'
NONE_TAG = '<none>:<none>'
NOP_PREFIX = '#(nop) '
options = {}
commands = []
;; # Default to -h if no arguments
ARGV << '-h' if ARGV.empty?
OptionParser.new do |opts|
opts.banner = "Usage: dockerfile-from-image.rb [options] <image_id>"
opts.on("-f", "--full-tree", "Generate Dockerfile for all parent layers") do |f|
options[:full] = f
end
opts.on_tail("-h", "--help", "Show this message") do
puts opts
exit
end
end.parse!
image_id = ARGV.pop
abort('Error: Must specify image ID or tag') unless image_id
;; # Collect all image tags into a hash keyed by layer ID.
;; # Used to look-up potential FROM targets.
tags = Docker::Image.all.each_with_object({}) do |image, hsh|
tag = image.info['RepoTags'].first
hsh[image.id] = tag unless tag == NONE_TAG
end
loop do
;; # If the current ID has a tag, render FROM instruction and break
;; # (unless this is the first command)
if !options[:full] && commands && tags.key?(image_id)
commands << "FROM #{tags[image_id]}"
break
end
begin
image = Docker::Image.get(image_id)
rescue Docker::Error::NotFoundError
abort('Error: specified image tag or ID could not be found')
end
cmd = image.info['ContainerConfig']['Cmd']
if cmd && cmd.size == 3
cmd = cmd.last
if cmd.start_with?(NOP_PREFIX)
commands << cmd.gsub(NOP_PREFIX,'').gsub(/^USER[ |\t]+\[(.*)\]/,'USER \1')
else
commands << "RUN #{cmd}".split.join(' ')
end
end
image_id = image.info['Parent']
break if image_id == ''
end
puts commands.reverse
17.17.21. image:jenkins/jenkins
- Parent images: openjdk:8-jdk
- GitHub
# latest LTS docker pull jenkins/jenkins:lts # Volume jenkins_home is created and holds Jenkins data directory includes plugins and configuration for upgrade docker run --name myjenkins -d -v jenkins_home:/var/jenkins_home -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts # Pass JVM parameters docker run --env JAVA_OPTS=-Dhudson.footerURL=http://mycompany.com # other options... # config logging mkdir data cat > data/log.properties <<EOF handlers=java.util.logging.ConsoleHandler jenkins.level=FINEST java.util.logging.ConsoleHandler.level=FINEST EOF docker run --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/log.properties" # other options... # JENKINS_OPTS # run Jenkins behind Nginx # mysite.com/jenkins docker run --env JENKINS_OPTS="--prefix=/jenkins" # https://wiki.jenkins.io/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy
17.17.22. image:ubuntu
# as the time of writing, these lines are equivalent docker pull ubuntu:latest docker pull ubuntu:18.04 docker pull ubuntu:bionic docker pull ubuntu:bionic-20181112 # The default variant for arm64 is currently v8 docker pull arm64v8/ubuntu:bionic # run and bash in. Refer to docker:run:privileged docker run -it --rm --privileged ubuntu:xenial /bin/bash
services: nodejs: image: ubuntu:latest tty: true
17.17.23. image:node
- https://github.com/nodejs/docker-node
- Available images
/usr/local/bin/docker-entrypoint.sh- Run command with node if the first argument contains a
-or is not a system command
- Run command with node if the first argument contains a
node- Use
bashto bash in
- Use
root- https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md
17.17.23.1. Run image directly
docker run -it --rm --name my-running-script \ -v "$PWD":/usr/src/app \ -w /usr/src/app \ node:8 \ node your-daemon-or-script.js # `node your-daemon-or-script.js` is a command run. Can be other things npm install && npm run build
.PHONY: enter enter: docker run -it --rm --name my-running-script \ -v `pwd`:/usr/src/app \ -w /usr/src/app \ -p 3001:3001 \ node:18.18 \ bash -c "npm i -g serve && bash"
17.17.23.2. Version
- node:boron
Latest version 6.x (LTS) Parent image: buildpack-deps:jessie Available: git, openssh-client
Use
docker logs -f container_nameto tail log - node:<version>
- 18.18
- may have variants e.g.
18.18.0 - (no term)
10.15.3
FROM --platform=linux/x86_64 node:10.15.3 RUN mkdir /app WORKDIR /app CMD ["node", "--version"]
node-build-image: docker build --force-rm --no-cache -t mynode:1.0 -f Dockerfile-node . npminstall: cd ../../relative/path; docker run -it --rm --platform linux/x86_64 --name mynodecontainer \ -v `pwd`:/app \ mynode:1.0 \ /bin/sh -c 'cd sass; npm install' npmrunbuild: cd ../../relative/path; docker run -it --rm --platform linux/x86_64 --name mynodecontainer \ -v `pwd`:/app \ manode:1.0 \ /bin/sh -c 'cd sass; npm run build'
17.17.23.3. Dockerfile
FROM node:16 # Place global NPM dependencies in a non-root user directory ENV NPM_CONFIG_PREFIX=/home/node/.npm-global # optionally if you want to run npm global bin without specifying path ENV PATH=$PATH:/home/node/.npm-global/bin # Install global NPM modules RUN npm install --global zapier-platform-cli
17.17.24. image:golang
golang:1.8 Parent image: buildpack-deps:jessie-scm Available: make Environment vars: GOPATH=/go GOROOT: usr/local/bin WORKDIR $GOPATH
docker run -it --rm --name testgo golang:1.8 /bin/bash -I to go in
Running exec hello inside container will exit the container
To download, compile and run and exit the container:
docker run --rm golang:1.8 sh -c "go get github.com/golang/example/hello/... && exec hello"
To download the compiled hello command to host directory tmp/bin:
~docker run –rm -v /tmp/bin:/go/bin golang:1.8 go get github.com/golang/example/hello…~
Then run /tmp/bin/hello where hello is the parent directory name
(For private repo, add -it for inputing username and password)
17.17.25. image:vimagick/scrapyd
https://hub.docker.com/r/vimagick/scrapyd/ https://github.com/vimagick/dockerfiles/tree/master/scrapyd
docker pull vimagick/scrapyd nano docker-compose.yml
version: '2' services: scrapy: image: vimagick/scrapyd command: bash volumes: - "./:/code" working_dir: /code restart: always
import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'http://quotes.toscrape.com/tag/humor/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').extract_first(), 'author': quote.xpath('span/small/text()').extract_first(), } next_page = response.css('li.next a::attr("href")').extract_first() if next_page is not None: yield response.follow(next_page, self.parse)
$ docker-compose run --rm scrapy >>> scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json >>> cat top-stackoverflow-questions.json >>> exit
17.17.26. image:yandex/clickhouse-server
- https://hub.docker.com/r/yandex/clickhouse-server
- Container exposes 8123 port for HTTP interface and 9000 port for native client
version: '3.5' services: madocker: # .. networks: - default clickhouse: container_name: myclickhouse image: yandex/clickhouse-server ports: - "8123:8123" # user is optional, 101:101 is clickhouse:clickhouse, default is root user: "101:101" volumes: # refer to init.sh - ./setupfiles/clickhouse/:/docker-entrypoint-initdb.d # mapping /var/lib/clickhouse does not work on Mac 2022-03-24 # - ./clickhouse:/var/lib/clickhouse # - ./clickhouse-log:/var/log/clickhouse-server # optional networks: - default
docker exec -it myclickhouse bash # inside the container, logon clickhouse-client # to logon using a password and default host clickhouse-client --password abc # to logon to any server clickhouse-client --host <HOSTNAME> \ --secure \ -- port 9440 \ --user <USERNAME> \ --password <PASSWORD>
17.17.26.1. Init script files
#!/bin/bash set -e clickhouse-client -n <<-'EOSQL' CREATE DATABASE IF NOT EXISTS malog; CREATE TABLE IF NOT EXISTS malog.accesslog(...); EOSQL
17.17.27. image:ngrok/ngrok
# docker-compose is run and it creates a default network and a service called madocker # docker-compose.yml is in folder devdocker, so the network devdocker_default is created # --host-header overwrites the request's Host header # -p 3000:4040 :: http://localhost:3000 is now a web UI for debugging docker run -it --rm -p 3000:4040 --net devdocker_default -e NGROK_AUTHTOKEN=MY_AUTH_TOKEN ngrok/ngrok http --host-header="localhost:80" madocker:80
18. Lando
All subdomains of lndo.site will be DNS publicly resolved to localhost/127.0.0.1 which requires no local /etc/hosts config.
18.1. Recipe: WordPress
cd /path/to/project
lando init
lando start
lando wp core download
- Lando creates database container with
- user: wordpress
- password: wordpress
- database: wordpress
Sample yaml
name: scef recipe: wordpress config: webroot: . php: '7.0' # default '7.2' via: nginx # default apache # add redis services: cache: type: redis:2.8 # may just be redis persist: true
For Redis, add the following in wp-config.php according to
lando infofor plugin wp-redis lando:service:cache$redis_server = array( 'host' => 'cache', 'port' => 6379, //'auth' => '12345', 'database' => 0, // Optionally use a specific numeric Redis database. Default is 0. );
Commands
# drop into a MySQL Shell lando mysql lando db-import <file> # run php commands lando php
18.2. Recipe: Pantheon
- https://docs.devwithlando.io/tutorials/pantheon.html
- https://github.com/lando/lando/tree/master/plugins/lando-pantheon/recipes/pantheon
- Stop
Docker for Windowsand install Lando withoutDocker for WindowsandGit for Windows git clonefrom Pantheon, go to the directory and runlando initand choose pantheon and a site to create.lando.ymlin project root dirlando terminus auth:login --machine-token=my-machine-token- If terminus login fail
lando ssh --user=rootand runterminus auth:login --machine-token=xxx
lando startlando pull --database=live --files=live --code=nonelando pull --database=none --code=none --files=live --rsyncwp cache flushorlando wp cache flush- May need to clear cache from wp-admin after
lando rebuildor restart - wp admin > Settings > clear Pantheon Page Cache
- May need to clear cache from wp-admin after
- Fully remove all lando containers and networks
lando stop && lando destroy && lando poweroffdocker rm -f $(docker ps -aq)docker network prune- Usually
lando stop && lando destroy && lando startis enough. Can also uselando rebuild
lando terminus wp mysite.live user listlando push -m "a message" --database=none --files=none- Lando creates database container with:
- user
- pantheon
- password
- pantheon
- database
- pantheon
- the appserver container has the following
.shfiles in folder/helpers. Their counter parts are in docker host~/.lando/services/config/pantheon/*.sh. Both of them are updated whenlando pullis run- pull.sh / push.sh
- pantheon.sh
- Add
--allow-rootfor wp commands in those files and run/helpers/pull.sh --database=dev --files=none --code=nonedirectly inside the container - Or add a bash script wrapper wp-cli:allow-root
apt-get update && apt-get install nano$_ENV['PANTHEON_ENVIRONMENT']islando
18.2.1. Push a new website to Live lando:pantheon:live
- On local Lando
wp search-replace '//myweb.com' '//dev-myweb.pantheonsite.io' --dry-runwp search-replace '//www.myweb.com' '//dev-myweb.pantheonsite.io' --dry-runwp search-replace '//myweb.lndo.site' '//dev-myweb.pantheonsite.io --dry-run'
- Minimize files in wp-content/uploads
- Lando push code, db and files
- FileZilla to push missing file to Pantheon Dev
- On Pantheon, click Initialize Test Environment
- click Initialize Live Environment
- Connect to Live db and check domains. Only
//live-myweb.pantheonsite.ioshould exist - Change domain DNS setting and Setup redirect pantheon:domain
- Setup SendGrid
- Enable scheduled backup for all environments
18.2.2. Varnish lando:pantheon:varnish
- See version
varnishd -V - Go to CLI
varnishd, thenstopstartquit - Search log
varnishlog -d -q "BerespStatus == 503" - Config files
/etc/varnish/conf.d/*.vcl
18.3. Recipe: Drupal 7
lando init # drupal 7 # get config information lando info # import db. # Can handle uncompressed, gzipped or zipped files lando db-import db.sql.gz lando composer lando db-export 'db-export.sql.gz' lando db-import 'db-import.sql.gz' lando drush # MySQL Shell lando mysql # For postgres lando psql # installed PHP extensions lando php -,
18.3.1. Landofile .lando.yml
name: my-app-name recipe: drupal7 config: webroot: . php: '5.6' # default php:7.3 # via: apache:2.4 # default. check with Lando apache service for the default version database: mariadb:10.3 # default: mysql:5.7. check with Lando mysql service for the default version drush: "*" # default latest. Or drush:^7 for PHP 5.3 # drush: "*" # latest # drush: ^7 # latest version of Drush 7 # drush: 8.1.15 # a specific version # xdebug: false # default. config: # custom config files. database: config/my-custom.cnf php: config/php.ini server: config/server.conf # for nginx vhosts: config/default.conf # for apache
18.3.2. settings.php
// `lando drush` may return incorrect url. `lando info` to get the URL with or without port $base_url = "http://mysite.lndo.site:PORT_IF_NEEDED"; $databases = array ( 'default' => array ( 'default' => array ( 'database' => 'drupal7', 'username' => 'drupal7', 'password' => 'drupal7', 'host' => 'database', 'port' => '3306', 'driver' => 'mysql', // for postgress: username:postgres, password:'', port:5432 ), ), );
18.4. Service: phpMyAdmin
name: mysite recipe: pantheon config: framework: wordpress site: mysite id: XXX-XXX proxy: pma: - pma.mysite.lndo.site services: pma: type: phpmyadmin hosts: - database
18.5. Service: MariaDB
- Pantheon recipe uses MariaDB
- Supported versions: 10.1 (default), 10.2, 10.3
docker exec -it projname_database_1 bash mysql
18.6. Command line
- List names of apps
lando list- Destroy docker containers inside the current app
lando destroy. After that, it requires pull database, Pantheon Terminus login again.- Spin down all lando related containers
lando poweroff
18.6.1. ssh
lando ssh [appname] [service]
blank appname is current app blank service is appserver
Bash into appserver :: lando ssh --user=root
18.6.2. db-import
lando db-export cs-devops/a.sql.gz lando db-import a.sql.gz lando db-import a.sql
18.7. SSL
- Enable
https://yourapp.lndo.site
As of Lando 3.0.0 RC:
# Add the Lando CA sudo cp -r ~/.lando/certs/lndo.site.pem /usr/local/share/ca-certificates/lndo.site.pem sudo update-ca-certificates # Remove Lando CA sudo rm -f /usr/local/share/ca-certificates/lndo.site.pem sudo update-ca-certificates --fresh
- Add
lndo.site.pemto Chrome - Settings > Manage Certificates > Authorities > Import, select and check all options
18.8. XDebug lando:xdebug
Just add xdebug: true to a recipe's config or add it under a php service
name: myapp recipe: pantheon config: framework: wordpress site: myapp id: abc-xyz-1234 xdebug: true conf: # optional: load a custom config file with XDebug settings php: .cs-devops/php.ini
lando rebuildif.lando.ymlis modified
If xdebug is not triggering, try appending url parameter XDEBUG_START_SESSION=LANDO
If it's still not triggered, try:
lando restart- Still not working, remove
xdebug: truelando restartand set xdebug using customphp.ini
xdebug.max_nesting_level = 256 xdebug.show_exception_trace = 0 xdebug.collect_params = 0 xdebug.remote_enable = 1 xdebug.remote_autostart = 1 xdebug.remote_host = YOUR HOST IP ADDRESS
Run lando info --deep | grep IPAddress to help discover the host ip address
18.9. TS: Same app name but different locations
In one folder, lando destroy then completely remove that folder.
In the new folder, lando destroy && lando poweroff and remove all docker containers and network prune. And run lando start again.
18.10. Uninstall
lando stop lando destroy lando poweroff docker rm -f $(docker ps -aq) # or only Lando containers docker rm -f $(docker ps --filter label=io.lando.container=TRUE --all -q) docker network prune -f
- Windows
- Add/Remove Program to uninstall Lando
- Ubuntu
sudo apt-get remove landoor use Software Center to uninstall
- Remove config
~/.lando- find config location
lando config | grep userConfRoot - (no term)
- Windows
C:\Users\myname\.lando
19. Jenkins
19.1. Install
Ubuntu or Debian-based
wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins
It will:
- Setup Jenkins as a daemon launched on start. See
/etc/init.d/jenkinsfor more details. - Create a jenkins user to run this service.
- Direct console log output to the file
/var/log/jenkins/jenkins.log. Check this file if you are troubleshooting Jenkins. - Populate
/etc/default/jenkinswith configuration parameters for the launch, e.g JENKINS_HOME - Set Jenkins to listen on port 8080. Access this port with your browser to start configuration.
- If your /etc/init.d/jenkins file fails to start Jenkins, edit the /etc/default/jenkins to replace the line -—HTTP_PORT=8080-— with -—HTTP_PORT=8081-— Here, "8081" was chosen but you can put another port available.
sudo /etc/init.d/jenkins restart
Usage: /etc/init.d/jenkins {start|stop|status|restart|force-reload}`
19.2. config
nano /etc/default/jenkins Disable DNS Multicast feature which will blow up jenkins log
Change this: JAVA_ARGS="-Djava.awt.headless=true" To this: JAVA_ARGS="-Djava.awt.headless=true -Dhudson.DNSMultiCast.disabled=true" Restart service jenkins restart
Or you can keep that feature on but just don't log
Manage Jenkins -> System Log -> Log Levels (on the left) Add the following entry:
Name: javax.jmdns Level: off
19.3. Builds
Always use sudo to run commands
20. Go
20.1. Basics, Command
Go is just an executable script file. Uninstall on Windows, go to Control Panel if you install using msi On Linux, uninstall by just replacing the bin folder
Install on Windows using zip, extract it to C:\go so C:\go\bin exists. Add C:\go\bin to PATH. GOPATH is C:\Users\username\go See phpstorm:golang
echo $GOPATH go version # godoc <package_name> godoc fmt gofmt main.go # -w write file gofmt -w main.go # Format all files in directory go fmt ./... # List environment variables go env # GOPATH :: e.g. "/root/go" on linux, where downloaded modules/packages are saved. # GOROOT = usr/local/go # GOARCH :: e.g. "arm64" # GOOS :: like "linux" # cat $GOROOT/src/go/build/syslist.go # Compile packages and dependencies based on 1 go file go build main.go # A package file name should be only lowercase letters # Folder structure in GOPATH directory # ./bin holds executables (compiled # ./pkg # ./src holds packages. In this folder, run go install # to compile and executables in ./bin # Change GOOS and GOARCH then go install again # Standard library including builtin package are under $GOROOT/src, e.g. $GOROOT/src/fmt # Or go online https://golang.org/pkg/fmt/
In Go Playground, the system time is always the same http://play.golang.org
20.1.1. go run -race *.go
-race is to detect data race ./compare.go
package main import (...) func f1 func f2
./main.go
package main
import (...)
func main() {
f1()
}
20.2. Syntax, Program Flow
Opening and closing brackets have to be in one line
// for ... {
// } else if x ==0 {
// } else {
Define variable in block
if x :=-42; x < 0 {
// x will only be available in the if block
}
Switch
import (
"math/rand"
"time"
)
rand.Seed(time.Now().Unix())
result := ""
switch dow := rand.Intn(6) + 1; dow {
// between 0 and 6. Result is between 1 and 7
case 1:
result = "Sunday!"
case 7:
result = "Saturday!"
default:
result = "Weekday!"
}
fmt.Println(result)
// Conditional expression in switch!
x := -42;
switch {
case x < 0:
result = "Less than zero"
case x == 0:
result = "Equal to zero"
default:
result = "Greater than zero"
}
// By default, it breaks and goes to the end of switch if there's one case matched.
// break is not necessary but `fallthrough` replaces `break` in an opposite way
for
sum = 0
for i := 0; i < 10; i++ {
sum += i
}
for sum < 1000 {
sum += sum
fmt.Println(sum)
if sum > 200 {
goto endofprogram
}
}
endofprogram : fmt.Println("end of program")
// Use `break` or `continue` statements just like other language
20.3. Standard Library of packages
20.3.1. builtin
20.3.1.1. new(), make()
new() :: Allocate but not initial memory. Results in 0 storage but return a memory address make() :: Allocate and initialize memory. Results in non-zeroed storage and return a memory address
Zeroed storage means it can't accept value
Initialize complex objects before adding values. Declarations without make() can cause a panic
m := make(map[string]int) m["key"] = 42 // map[key:42]
Memory is deallocated by garbage collector (GC).
GC clears objects that are out of scope or set to nil
20.3.2. fmt
20.3.2.1. Output
package main
import (
fmt
)
func main() {
str1 := "one"
str2 := "two"
aNumber := 42
// var aNumber int
// default aNumber is zero
// Implicit typing using :=
isTrue := true
// Explicit: const anInteger int = 42
// Implicit const aString = ".."
fmt.Println(str1, str2) // "one two"
stringLength, err := fmt.Println(str1, str2)
// function can return multiple values, a string and an error
if err == nil {
fmt.Println("String Length", stringLength)
}
// Without the if err == nil, it will throw an error because error is not defined
// Use _ to ignore the err variable even if it's not returned
stringLength, _ := fmt.Println(str1, str2)
fmt.Println("String Length", stringLength)
fmt.Printf("Value of aNumber: %v\n", aNumber)
fmt.Printf("Value of isTrue: %v\n", isTrue)
fmt.Printf("Value of aNumber as float: %.2f\n", float64(aNumber)) // 42.00
fmt.Printf("Data types: %T, %T, %T, %T",
str1, str2, aNumber, isTrue
)
// string, string, int, bool
// Return a string
myString := fmt.Sprintf("data types: %T, %T, %T, %T", str1, str2, aNumber, isTrue)
fmt.Print(myString)
}
%v :: only values %+v :: add field names when printing struct %#v :: add field names and name of the struct type when printing struct %T :: type %% :: literal %
Sprintf(format string, a …interface{}) string :: format and return a string
Use this to convert a struct into string str := fmt.Sprintf("%#v", val)
Fprintf(w io.Writer, format string, a …interface{}) (n int, err error)
20.3.2.2. Input
import ("fmt")
func main() {
var s1,s2 string
fmt.Scanln(&s1, &s2)
// Scan one line from terminal input, demilited by space
fmt.Println(s1, s2)
}
import (
"fmt"
"bufio"
"os"
"strconv"
"strings"
)
func main() {
// := is to declare and assign value to a new variable
reader := bufio.NewReader(os.stdin)
fmt.Print("Enter text: ")
str, _ := reader.ReadString('\n')
// Single quote means it's a byte not a string
fmt.Print(str)
// one two three
fmt.Print("Enter text: ")
str, _ = reader.ReadString('\n')
f, err := strconv.ParseFloat(strings.TrimSpace(str), 64)
if err != nill {
fmt.Println(err)
} else {
fmt.Println("Value of number: ", f);
}
}
20.3.3. strings
strings.ToUpper(str) strings.Title(str) / Initial capped strings.EqualFold(str1, str2) / case-insensitive compare / default == is case-sensitive strings.Contains(str, "exp") / true/false
20.3.4. math
import (
"fmt"
"math/big"
"math"
)
i1, i2, i3 := 12, 45, 68
intSum := i1 + i2 + i3 // Good!
f1, f2, f3 := 23.5, 65.1, 76.3
floatSum := f1 + f2 + f3 // Not good
var b1, b2, b3, bigSum big.Float
b1.SetFloat64(23.5)
b2.SetFloat64(65.1)
b3.SetFloat64(76.3)
bigSum.Add(&b1, &b2).Add(&bigSum, &b3)
fmt.Printf("BigSum = %.10g\n", &bigSum)
circleradius := 15.5
circumference := circleRadius * math.Pi
fmt.Printf("Circumference: %.2f\n", circumference)
20.3.5. time
import (
"fmt"
"time"
)
t := time.Date(2009, time.November, 10, 23, 0,0,0, time.UTC)
now := time.Now()
fmt.Printf("Go launched at %s\n", t)
now.Month()
now.Day()
now.Weekday()
tomorrow := now.AddDate(0,0,1)
longFormat := "Monday, January 2, 2006"
fmt.Println("Tomorrow is ", tomorrow.Format(longFormat))
shortFormat := "1/2/06"
t := time.Now()
fmt.Printf("%d %02d %02d %02d %02d %02d\n",
t.Year(), t.Month(), t.Day(),
t.Hour(), t.Minute(), t.Second())
start := time.Now() // ... end := time.Now() delta := end.Sub(start) // Or fmt.Println(time.Since(start).Seconds())
20.3.6. sort
20.3.7. io, os, path
import (
"io"
"os"
"io/ioutil"
"path/filepath"
)
func main() {
content := "Hello from Go!"
file, err :=os.Create("./fromString.txt")
checkError(err)
defer file.Close()
ln, err := io.WriteString(file, content)
checkError(err)
fmt.Printf("All done with file of %v characters", ln)
// convert a string to binary
bytes := []byte(content)
// 0644 is file permission
ioutil.WriteFile("./fromBytes.txt", bytes, 0644)
// Read a file
fileName := "./hello.txt"
// read as binary
content, err := ioutil.ReadFile(fileName)
checkError(err)
// binary/byte
fmt.Println(content)
result := string(content)
fmt.Println(result)
// Reader a directory
root, _ := filepath.Abs(".")
fmt.Println(root)
err := filepath.Walk(root, processPath)
checkError(err)
}
func checkError(err error) {
if err != nil {
// Stop if there's an error
panic(err)
}
}
func processPath(path string, info os.Fileinfo, err error) error {
if err != nil {
return err
}
// not root
if path != "." {
if info.IsDir() {
fmt.Println("Directory:", path)
} else {
fmt.Println("File:", path)
}
}
return nil
}
Print OS environments
// To set a key/value pair, use `os.Setenv`. To get a
// value for a key, use `os.Getenv`. This will return
// an empty string if the key isn't present in the
// environment.
os.Setenv("FOO", "1")
fmt.Println("FOO:", os.Getenv("FOO"))
fmt.Println("BAR:", os.Getenv("BAR"))
// Use `os.Environ` to list all key/value pairs in the
// environment. This returns a slice of strings in the
// form `KEY=value`. You can `strings.Split` them to
// get the key and value. Here we print all the keys.
fmt.Println()
for _, e := range os.Environ() {
pair := strings.Split(e, "=")
fmt.Println(pair[0])
}
20.3.8. path/filepath
filepath.Walk("../images/", func(path string, info os.FileInfo, err error) {
if info.IsDir() {
return nil
}
fmt.Println(path)
return nil
})
20.3.9. encoding/json, encoding/csv
Refer to golang:net/http:json
json.NewDecoder(r io.Reader) *Decoder
json.Unmarshal(data []byte, v interface{}) error
// convert any json string to struct
import ("encoding/json" "fmt" "reflect")
var input = `
{
"response_type": "in_channel",
"text": "hello text",
"attachments": [
{ "text": "attachment text" }
]
}
`
var val map[string]interface{}
if err := json.Unmarshal([]byte(input), $val); err != nil {
panic(err)
}
fmt.Println(val)
// map[response_type:in_channel text:hello text attachments:[map[text:attachment text]]]
for k, v := range val {
fmt.Println(k, reflect.TypeOf(v))
}
import (
"os"
"encoding/csv"
)
f, err := os.Open("../abc.txt")
checkError(err)
defer f.Close()
rdr := csv.NewReader(f)
rdr.Comma = '\t'
rdr.TrimLeadingSpace = true
rows, err := rdr.ReadAll()
checkError(err)
for i, row := range rows {
fmt.Println(row[i])
}
20.3.10. log
Output to stdout
import ( "log" ) log.Println(string(body))
Write to os.Stdout os.Stderr
import (
"io"
"io/ioutil"
"log"
"os"
)
var (
Trace *log.Logger
Info *log.Logger
Warning *log.Logger
Error *log.Logger
)
func Init(
traceHandle io.Writer,
infoHandle io.Writer,
warningHandle io.Writer,
errorHandle io.Writer) {
Trace = log.New(traceHandle,
"TRACE: ",
log.Ldate|log.Ltime|log.Lshortfile)
Info = log.New(infoHandle,
"INFO: ",
log.Ldate|log.Ltime|log.Lshortfile)
Warning = log.New(warningHandle,
"WARNING: ",
log.Ldate|log.Ltime|log.Lshortfile)
Error = log.New(errorHandle,
"ERROR: ",
log.Ldate|log.Ltime|log.Lshortfile)
// log.New(out io.Writer, prefix string, flag int)
}
func main() {
// system devices that support io.Writer interface
Init(ioutil.Discard, os.Stdout, os.Stdout, os.Stderr)
// write to stdout
Info.Pringln("...")
}
Write to file
file, err := os.OpenFile("file.txt", os.O_CREATE|os.WRONLY|os.O_APPEND, 0666)
if err != nil {
log.Fatalln("Failed to open log file", output, ":", err)
}
multi := io.MultiWriter(file, os.Stdout)
MyFile := log.New(multi,
"PREFIX: ",
log.Ldate|log.Ltime|log.shortfile)
log.Fatalln("…") same as Println() but followed by a call to os.Exit(1) log.Panicln() same as Println() but followed by panic()
20.3.11. os/exec
20.3.11.1. Basics
import (
"fmt"
"os"
"os/exec"
)
func main() {
cmd := "curl"
args := []string{"https://..."}
/* if command contains wildecard *
exec.Command("/bin/sh", "-c", "mv ./source_dir/* ./dest_dir")
*/
// Simply run
if err := exec.Command(cmd, args...).Run(); err != nil {
fmt.Println(os.Stderr, err)
os.Exit(1)
}
fmt.Println("Success!")
// Simply run.
// Run and catch STDOUT
var (
cmdOut []byte
err error
)
if cmdOut, err = exec.Command(cmd, args...).Output(); err != nil {
fmt.Fprintln(os.Stderr, "An error: ", err)
os.Exit(1)
}
result := string(cmdOut)
// Run and catch STDOUT
// Catch output line by line as a stream
// Use cmd.Start() and cmd.Wait()
// Catch stdout by adding pipe
cmd2 := exec.Command(cmd, args...)
cmdReader, err := cmd2.StdoutPipe()
if err != nil {
fmt.Fprintln(os.Stderr, "Error creating StdoutPipe for cmd", err)
os.Exit(1)
}
// import "bufio"
scanner := bufio.NewScanner(cmdReader)
go func() {
for scanner.Scan() {
fmt.Printf("Output header | %s\n", scanner.Text())
}
}()
err = cmd2.Start()
if err != nil {
fmt.Fprintln(os.Stderr, "Error starting Cmd", err)
}
err = cmd2.Wait()
if err != nil {
fmt.Fprintln(os.Stderr, "Error waiting for Cmd", err)
os.Exit(1)
}
// Catch output line by line as a stream.
}
20.3.11.2. SSH command
type SSHCommander struct {
User string
IP string
}
func (s *SSHCommander) Command(cmd ...string) *exec.Cmd {
arg := append(
[]string{
fmt.Sprintf("%s@%s", s.User, s.IP),
},
cmd...,
)
return exec.Command("ssh", arg...)
}
func main() {
commander := SSHCommander{"root", "50.112.213.24"}
cmd := []string{
"apt-get",
"install",
"-y",
"jq",
"golang-go",
"nginx",
}
if err := commander.Command(cmd...); err != nil {
fmt.Fprintln(os.Stderr, "There was an error running SSH command: ", err)
os.Exit(1)
}
}
20.3.11.3. Cmd
Customize a command before it gets run by Run, Output or CombinedOutput methods
cmd := "git"
args := []string{"status"}
cmd2 := exec.Command(cmd, arg...)
// Working directory otherwise is the process's current dir
cmd2.Dir = "/home/user/"
20.3.12. net/http
20.3.12.1. HTTP request
resp, err := http.Get("http://example.com/")
...
resp, err := http.Post("http://example.com/upload", "image/jpeg", &buf)
...
resp, err := http.PostForm("http://example.com/form",
url.Values{"key": {"Value"}, "id": {"123"}})
import (
"io/ioutil"
"net/http"
"encoding/json"
"strings"
"math/big"
)
type Tour struct {
Name, Price string
}
func main() {
url := "http://..."
resp, err := http.Get(url)
checkError(err)
fmt.Printf("Response type: %T\n", resp)
// Important!!!
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.body)
checkError(err)
content := string(bytes)
fmt.Pritnln(content)
tours := toursFromJson(content)
fmt.Println(tours)
for _, tour := range tours {
// price is a string and convert to float
price, _, _ := big.ParseFloat(tour.Price, 10, 2, big.ToZero)
fmt.Printf("%v $%.2f\n", tour.Name, price)
}
}
func toursFromJson(content string) []Tour {
tours := make([]Tour, 0, 20)
decoder := json.NewDecoder(strings.NewReader(content))
_, err := decoder.Token()
checkError(err)
var tour Tour
for decoder.More() {
err := decoder.Decode(&tour)
checkError(err)
tours = append(tours, tour)
}
return tours
}
HTTP POST json
tmp := `{"text": "delayed response", "attachments": [{ "text": "test"}]}`
// or
// tmp, err := json.Marshal(myStruct)
req := bytes.NewBuffer([]byte(tmp))
resp, err := http.Post(sc.ResponseURL, "application/json", req)
checkError(err)
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
20.3.12.2. HTTP server
- Use http.Handle
import ( "fmt" "net/http" "html/template" ) // default Handler type Hello struct{} // Implement handler that is an interface func (h Hello) ServeHTTP(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "<h1>hello</h1>") } // func Handle(pattern string, handler Handler) // type Handler interface { // ServeHTTP(resp, *request) // } func main() { var h Hello err := http.ListenAndServe(":4000", h) // addr string, handler Handler // if handler is nil, DefaultServeMux is used // or localhost:4000 (only localhost:4000 listens. multiple domain, non localhost, can be resolved to the same server) checkError(err) } - Use http.HandleFunc
// HandleFunc add handlers to DefaultServeMux http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "hello html") }) fmt.Println(http.ListenAndServe(":4000", h)) checkError(err) - Template
import "html/template" type Page struct { Name string } func main() { templates := template.Must(template.ParseFiles("templates/index.html")) http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { p := Page{Name: "Gopher"} if err := templates.ExecuteTemplate(w, "index.html", p); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } }) }index.html
{{.Name}}If url.name is defined, then that will be displayed
- Read POST data
- Content-Type json golang:net/http:json
type test_struct struct { Test string IsNew bool // JSON booleans Age float64 // JSON numbers myArray []interface{} // JSON arrays myObj map[string]interface{} // JSON objects isNull nil // JSON null } func test(w http.ResponseWriter, r *http.Request) { // Read body as a whole body, err := ioutil.ReadAll(r.Body) checkError(err) log.Println(string(body)) //decoder := json.NewDecoder(r.Body) var t test_struct err = json.Unmarshal(body, &t) checkError(err) fmt.Fprintf(w, t.Test) // Get Header if a := r.Header.Get("x-hub-signature"); len(a) == 0 { panic() } } func main() { http.HandleFunc("/test", test) }See 15.20.1.5
- Content-Type:application/x-www-form-urlencoded
r.ParseForm() r.Form.Get("key") res := r.FormValue("<your param name>") - Return Json as application/json
func testjson(w http.ResponseWriter, r *http.Request) { body, err := ioutil.ReadAll(r.Body) checkError(err) type slackResponse struct { ResponseType string `json:"response_type"` Text string `json:"text"` Attachments []map[string]string `json:"attachments"` } a2 := []map[string]string{map[string]string{"text": string(body)}} resp := slackResponse{"in_channel", a2} js, err := json.Marshal(resp) checkError(err) w.Header().Set("Content-Type", "application/json") w.Write(js) /* Or you can just output a json string var p = `{ "response_type": "in_channel", "attachments": [ { "text": "%s" } ] }` resp := fmt.Sprintf(p, string(body)) w.Header().Set("Content-Type", "application/json") fmt.Fprintf(w, resp) */ } func main() { http.HandleFunc("/json", testjson) }More complicated value `attachments`
// this is the json to construct // {"response_type":"in_channel","text":"hello text","attachments":[{"text":"attachment text"},{"text2":{"name":"hello"}}]} type Profile struct { Response_type string `json:"response_type"` Text string `json:"text"` Attachments []interface{} `json:"attachments"` } func main() { a2 := make([]interface{}, 2) a2[0] = map[string]string{"text": "attachment text"} a2[1] = map[string]map[string]string{"text2": map[string]string{"name": "hello"}} profile := Profile{"in_channel", "hello text", a2} js, err := json.Marshal(profile) checkError(err) fmt.Println(string(js)) str := fmt.Sprintf("%#v", profile) fmt.Println(str) }
- Content-Type json golang:net/http:json
20.4. Custom package, Third Party Package
For third-party packages, godoc.org and github.com/avelino/awesome-go https://www.lynda.com/Go-tutorials/What-you-should-know-before-watching-course/439416/472998-4.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3ago+language%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2 $GOPATH/src/mypkg
package main
// define a package name.
// main() is the starting point in go run
func main() {
n1, l1 := FullName("Zaphod", "Beeblerox")
}
// captital P means this method is public instead of private in this package
func Println() {
}
func addValues(value1 int, value2 int) int {
// can also be value1, value2 int
return value1 + value2
}
func addAllValues(values ...int) int {
sum := 0
for i := range values {
sum += values[i]
}
return sum
}
func FullName(f, l string) (string, int) {
full := f + " " + l
length := len(full)
return full, length
}
func FullNameNakedReturn(f, l string) (full string, length int) {
// The returned variables have already been declared
// Don't use :=
full = f + " " + l
length = len(full)
return
}
;; # mypkg.Println(...)
import (
// github.com/username/reponame/foldername/example.go
"github.com/username/reponame/foldername"
// alias
stdimage "image"
)
All packages and codes should be under one go workspace which the folder at GOPATH. Assuming GOPATH is ~/go in Windows installation.
cd ~/go mkdir -p src/github.com git clone ../gitusername/myreponame ~/go/src/github.com/gitusername/reponame/pkg1/pkg1.go ~/go/src/github.com/gitusername/reponame/pkg2/pkg2.go ;; # build all necessary packages go get github.com/gitusername/reponame/... ;; # ~/go/bin/pkg1 (executable) ;; # ~/go/bin/pkg2 (executable) ;; # Build one package go get github.com/gitusername/reponame/pkg1/... ;; # ~/go/bin/pkg1 (executable) ;; # At ~/go/src/github.com/gitusername/reponame, run this to see if there's compile error go build ... ;; # At ~/go/src/github.com/gitusername/reponame, run this to format all .go files go fmt ./...
20.5. Types
20.5.1. Basic
float32 float54 complex64 complex128 Integers: int8 int16 int32 int64 only positive numbers and zero (unsigned): uint8 uint16 uint32 uint64 byte uint int uintptr
Data collections: Arrays Slices Maps Structs
Language organization: Functions Interfaces Channels
Data management Pointers
Type conversion
var int1 int = 5
var float1 float64 = 42
var s := `{ "votes": { "option": "3"} }`
sum := float64(int1) + float1
// convert string to byte
bytevar := []byte(s)
// convert string array to string
stringArray := []string{"Hello","world","!"}
justString := strings.Join(stringArray," ")
// Convert string to float golang:strconv
f, err := strconv.ParseFloat(strings.TrimSpace(str), 64)
// Delcare only for multiple variables
var (
count int
sum float64
)
var a, b string
a, b := "", ""
20.5.2. array
Array can't be sorted or resized
var colors [3]string
colors[0] = "Red"
colors[1] = "Green"
fmt.Println(colors)
// [Red Green]
fmt.Println(colors[1])
// Green
// Define in one line
var numbers = [5]int{5,3,1,2,4}
// Array length
// len(colors)
for i := 0; i < len(colors); i++ {
fmt.Println(colors[i])
}
for i := range colors {
fmt.Pritnln(colors[i])
}
20.5.3. slice
slice is built on top of array
var colors = []string{"Red", "Green", "Blue"}
fmt.Println(colors)
// The same as array
// Add item to end of slice
colors = append(colors, "Purple")
// Remove the first item in slice
colors = append(colors[1:len(colors)])
// Below is the same to remove the firs item in slice
// as the default on the right of : is the length of the slice
colors = append(colors[1:])
// Remove the last item in slice
// the default on the left of : is 0
colors = append(colors[:len(colors)-1])
// Declare a slice using make
numbers := make([]int, 5, 10)
// 5 is inital size, 10 is capacity (optional)
// Current capacity cap(numbers)
// Define 5 values, then add another item,
numbers = append(numbers, 123)
// cap(numbers) is 2+10 = 12
// import("sort")
sort.Ints(numbers) // ascending
20.5.4. map
Map is an unorderd collection of key-value pairs
states := make(map[string]string)
// empty: map[]
states["WA"] = "Washington"
states["OR"] = "Oregon"
// map[WA:Washington OR:Oregon]
delete(states, "OR")
for k, v := range states {
fmt.Printf("%v: %v\n", k, v)
}
// create a slice to hold keys of a map
keys :=make([]string, len(states))
i := 0
for k := range states {
keys[i] = k
i++
}
sort.Strings(keys)
for i := range keys {
fmt.Println(states[keys[i]])
}
20.5.5. struct, interface
Struct is a data structure. Similar to Java's classes: encapsulate data and methods. No inheritance.
Every value in Go is an instance of a type, and every type is an implementation of at least one interface, the interface without any methods.
You may see function like this.
func Errorf(format string, a ...interface{}) {}
// ...interface{} means it can be multiple values of all types
type Dog struct {
Breed string
Weight int
Sound string
}
// Define method for a Struct
func (d Dog) Speak() {
// d is a reference (copy) of the original object
// changing d here doesn't affect the variable in global scope
fmt.Println(d.Sound)
}
func (d *Dog) SpeakThreeTimes() {
// d is a point to the original object
// Changing d here also changes the variable in global scope
d.Sound = fmt.Sprintf("%v! %v! %v!", d.Sound, d.Sound, d.Sound)
fmt.Println(d.Sound)
}
// Define an interface
type Animal interface {
SpeakWord() string
}
func (d Dog) Speak() string {
return "Woof!"
}
// Attach interface to other class
type Cat struct {
// ...
}
func (c Cat) Speak() string {
return "Meow!"
}
func main() {
poodle := Dog{"Poodle", 34, "Woof"}
// fmt.Println(poodle)
// {Poodle, 34, Woof}
// dump field name and values
fmt.Printf("%+v\n", poodle)
// {Breed:Poodle Weight:34 Sound:Woof}
poodle.Sound = "Arf"
poodle.Speak()
poodle2 := Animal(Dog{})
// This dog has an interface!
// Or this dog is converted to interface Animal
fmt.Println(poodle2)
// Attach interface to multiple objects of different types
animals := []Animal{Dog{}, Cat{}}
for _, animal := range animals {
// _ means to throw away the index
fmt.Println(animal.Speak())
}
}
20.5.6. pointer
// craete a pointer
var p *int
if p != nil {
fmt.Println(*p)
} else {
fmt.Println("p is nil")
}
var v int = 42
p = &v
// explict
var value1 float64 = 42.13
pointer1 := &value1
fmt.Println(*p)
*pointer1 = *pointer1 / 42
// both value1
fmt.Println(*pointer, value1)
20.5.7. error
import "errors"
myError := errors.New("My error string")
fmt.Println(myError)
// str := myError.Error()
attendance := map[string]bool{
"Ann": true,
"Mike": true}
attended, ok := attendance["Mike"]
if ok {
// do things
} else {
// return error
}
20.6. Concurrency
// Avoid race condition
var mu sync.Mutex
var wg sync.WaitGroup
wg.Add(len(paths))
for _, path := range paths {
go func(path string) {
pixels := getPixels(path)
mu.Lock()
{
images = append(images, pixels)
}
mu.Unlock()
wg.Done()
}(path)
}
20.7. Structure
;; # $GOPATH ./src/libraries ./src/palindrome ./src/stringutil ;; # Go to the directory that has a go file that has main() function ;; # and run go install
Defer defer keyword only works within a function Deferred statements are FILO and are run at the end of the current function, rather than waiting for the entire application Good for disconnect database
defer fmt.Println("Statement 1")
defer fmt.Println("Statement 2")
defer fmt.Println("Statement 3")
defer fmt.Println("Undeferred statement")
// Undeferred statement, 3, 2, 1
20.8. Shrink compiled binary file size
;; # same as go get, go install go build -ldflags="-s -w" cmd/go
It strips the DWARF tables needed for debugging, but keeps the annotations needed for stack traces (also panic as well)
Then use linux:upx to further compress it.
20.9. Project
20.9.1. GitHub Webhook go:github-webhook
#+NAME main.go
package main
import (
"fmt"
"net/http"
"io/ioutil"
"os"
"os/exec"
"log"
"crypto/hmac"
"crypto/sha1"
"encoding/hex"
"encoding/json"
"errors"
"strings"
)
type test_struct struct {
Repository map[string]interface{}
}
type GitHubHookContext struct {
Signature string
Event string
Id string
Payload []byte
}
type CIFolder struct {
repo string
Path string
}
func checkError(err error) {
if err != nil {
panic(err)
}
}
func signBody(secret, body []byte) []byte {
computed := hmac.New(sha1.New, secret)
computed.Write(body)
return []byte(computed.Sum(nil))
}
func verifySecret(secret []byte, signature string, body []byte) bool {
const signaturePrefix = "sha1="
const signatureLength = 45 // len(SignaturePrefix) + len(hex(sha1))
if len(signature) != signatureLength || !strings.HasPrefix(signature, signaturePrefix) {
return false
}
actual := make([]byte, 20)
hex.Decode(actual, []byte(signature[5:]))
return hmac.Equal(signBody(secret, body), actual)
}
func ParseGitHubHookContext(secret []byte, r *http.Request) (*GitHubHookContext, error) {
hc := GitHubHookContext{}
if hc.Signature = r.Header.Get("x-hub-signature"); len(hc.Signature) == 0 {
return nil, errors.New("No signature!")
}
log.Println("Signature: ", hc.Signature)
if hc.Event = r.Header.Get("x-github-event"); len(hc.Event) == 0 {
return nil, errors.New("No event!")
}
log.Println("Event: ", hc.Event)
if hc.Id = r.Header.Get("x-github-delivery"); len(hc.Id) == 0 {
return nil, errors.New("No event Id!")
}
log.Println("ID: ", hc.Id)
body, err := ioutil.ReadAll(r.Body)
if err != nil {
return nil, err
}
if !verifySecret(secret, hc.Signature, body) {
return nil, errors.New("Invalid signature")
}
hc.Payload = body
return &hc, nil
}
func pullGitHub(path string) {
var args []string
args = []string{"reset", "--hard"}
runCommands(path, "git", args, "git reset error: ")
args = []string{"clean", "-df"}
runCommands(path, "git", args, "git clean error: ")
args = []string{"checkout", "master"}
runCommands(path, "git", args, "git checkout master error: ")
args = []string{"pull"}
runCommands(path, "git", args, "git pull error: ")
}
func runCommands(path string, cmd string, args []string, msg string) {
var (
cmdOut []byte
err error
)
outputArgs := strings.Join(args, " ")
log.Println("Running ", cmd, outputArgs)
actual := exec.Command(cmd, args...)
actual.Dir = path
if cmdOut, err = actual.Output(); err != nil {
log.Println(os.Stderr, msg, err)
os.Exit(1)
}
log.Println(string(cmdOut))
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
secret := os.Getenv("GB_WEBHOOK_KEY")
hc, err := ParseGitHubHookContext([]byte(secret), r)
if err != nil {
log.Printf("Failed processing hook! ('%s')", err)
panic(err)
} else {
if hc.Event == "push" {
//log.Println(string(hc.Payload))
var t test_struct
err = json.Unmarshal(hc.Payload, &t)
checkError(err)
switch reponame := t.Repository["full_name"]; reponame {
case "NewcomDev/nodejs":
pullGitHub("/home/newcom/www/nodejs-dev")
}
}
}
fmt.Fprintf(w, "hello html")
//out := fmt.Sprintf("%#v", r.URL)
//fmt.Fprintf(w, out)
})
fmt.Println(http.ListenAndServe(":49162", nil))
}
sudo docker run -it --rm --name testgo -v /home/newcom/digitalocean/docker/newcomgo:/go golang:1.8 /bin/bash -I # Inside Golang go build main.go # Go to project folder ./main
20.10. To Learn
Concurrency in Go Goroutine
- a lightweight synchronized thread managed by the runtime
Channel
- a typed conduit for msgs between goroutines
Select
- Lets a goroutine wait for multiple communication operations
Web Frameworks (REST)
- Beego, Martini, Gorilla, Gocraft, Revel
Database Drivers
- Standard db functions: database/sql
- Driver interfaces: database/sql/driver
- https://github.com/golang/go/wiki/SQLDrivers
21. Vagrant
21.1. Install
- Install Oracle VM Virtualbox first
- https://www.virtualbox.org/wiki/Downloads
- VirtualBox Oracle VM VirtualBox Extension Pack is needed
- https://www.vagrantup.com/docs/virtualbox/
- (no term)
- VirtualBox version 5.0.x and 5.1.x are supported by Vagrant 1.x
- (no term)
- Simply install the latest version of VirtualBox to upgrade
- https://www.vagrantup.com/downloads.html
- If Oracle VM VirtualBox Extension Pack is previously installed, upgrading the VB core will also auto update it
- (no term)
TS:
Error: "VT-x/AMD-V hardware acceleration is not available"- On Windows, check if
Hyper-VorWindows Hypervisor Platformis on by going to Windows Features
- Run
optionalfeatures.exeto go directly to Windows Features - Turn it off. As Windows takes over the hardware acceleration
- Then reboot and change UEFI or BIOS setting
- name might be "Intel VT-x, VT-d, Virtualization Extensions, Vanderpool" under Chipset, Northbridge or CPU configuration
- On Windows, check if
- (no term)
- Install vagrant:plugin
- Hypervisor
- software, firmware or hardware that creates and runs virtual machines. May be called virtualization
- Hyper-V
- Windows' hypervisor
Docker for Windowsrequires Hyper-V on Windows- Windows Subsystem for Linux v2 (WSL2) supports docker using Hyper-V even on Windows Home edition
- Virtualbox hypervisor
- does not work with Windows having Hyper_V enabled
- Inside Virtualbox, cannot have another virtualization
vagrant --version # 1.x where vagrant # download image and create Vagrantfile vagrant init hashicorp/precise64 # Ubuntu 18.04 vagrant init hashicorp/bionic64 vagrant up && vagrant ssh
21.2. Vagrant VM and VirtualBox
# Status of all Vagrant VM's vagrant global-status --prune # List all running VirtualBox VMs # vboxmanage.exe is not in PATH, change directory cd /c/Program Files/Oracle/VirtualBox ./VBoxManage.exe list runningvms # Show a VM info ./VBoxManage.exe showvminfo myvm1 # VirtualBox version ./VBoxManage.exe --version
- Forward ports after Vagrant has started in VirtualBox
- Select the running VM in VirtualBox, Settings > Network > Port Forwarding, add e.g. forward
127.0.0.1:2223to10.0.2.15:22
- Select the running VM in VirtualBox, Settings > Network > Port Forwarding, add e.g. forward
- By default, VirtualBox does not enable symlink for shared folders for hosts that don't support symlink
- Hosts such as Windows
- On Windows, enable 3
- Run this on Windows
./VBoxManage.exe setextradata livagrantdev VBoxInternal2/SharedFoldersEnableSymlinksCreate/vagrant 1 vagrant halt,vagrant up- Still not possible to create symlinks
ln -s path/to/source path/to/symlinkinside Vagrant but any symlinks created on Windows usingmklinkwork as intended on Vagrant
21.3. Frequent commands
# force to run provision vagrant up --provision # or vagrant reload --provision # remove the current vagrant VM vagrant destroy -f # shut down and boot up vagrant halt vagrant up # Hibernate vagrant suspend # Shut down and up vagrant reload # current vagrant vm running status vagrant status # Forward ports after Vagrant has started. Forward 7000 on host to 5432 on Vagrant VM vagrant ssh -- -L 7000:localhost:5432
Connect from Vagrant to Host using Host IP
# get the default gateway netstat -rn # Usually, it is 10.0.2.2
# Show current vagrant vm ssh config vagrant ssh-config # Usually it's 127.0.0.1:2222, notice the port may change as vagrant will auto select port if it's in use # save to ~/.ssh/config, make it possible to use ssh directly vagrant ssh-config --host myvagrant >> ~/.ssh/config ssh myvagrant
21.4. Vagrantfile
# -*- mode: ruby -*- # vi: set ft=ruby : ENV["LC_ALL"] = "en_US.UTF-8" # In case host couldn't pass locale into vagrant hostname = livagrantdev cpus = 2 memory = 2048 @importmysql = <<SCRIPT echo "docker run -it --rm -v /vagrant/mydbdump:/tmp/mydbdump"\ " --link wordpressdb:wpdb mysql:5.7.9 sh -c"\ " 'exec mysql"\ ' -h"$WPDB_PORT_3306_TCP_ADDR" -P"$WPDB_PORT_3306_TCP_PORT"'\ ' -uroot -p"$WPDB_ENV_MYSQL_ROOT_PASSWORD"'\ ' wordpress < /tmp/mydbdump/pantheon_db.sql'\ "'" >> /home/vagrant/import_db.sh chmod +x /home/vagrant/import_db.sh SCRIPT Vagrant.configure(2) do |config| # config.vm config.vm.box = username/boxname # optional # config.vm.box_version = "1.1.0" # config.vm.box_url = "http://" # hostname, appears when you vagrant ssh - vagrant@yourhostname config.vm.hostname = hostname # network config.vm.network "private_network", ip: "192.168.33.10" # VMs in the the private network (VirtualBox) can communicate # Give a static IP config.vm.network "forwarded_port", guest: 22, host: 2222, id: "ssh" config.vm.network "forwarded_port", guest: 80, host: 8080, auto_correct: true # On local host (windows), 192.168.33.10:8080 # Synced folder # default /vagrant # Create a root folder (readonly) in guest machine to hold Dockerfiles for docker provisioner # Create that folder if it doesn't exist in guest machine config.vm.synced_folder "./build/docker", "/docker_builds", create: true, mount_options: ["ro"] config.vm.provider "virtualbox" do |vb| vb.memory = memory vb.cpus = cpus vb.name = hostname vb.gui = true # enable GUI on VirtualBox # The name appears on VirtualBox GUI # Can override any settings config.vm, config.ssh, config.winrm and config.vagrant end # Provision # Upload File using SSH user (vagrant) config.vm.provision "file", source: "~/.gitconfig", destination ".gitconfig" # In line Shell config.vm.provision "shell", inline: @importmysql # Path to script file Shell # config.vm.provision "shell", # Provisioner docker - install docker on the VM and pull images! config.vm.provision "install_docker", type: "docker" do |d| d.pull_images "mysql:5.7.9" d.pull_images "wordpress" # Need to create a root folder in guest machine to pull Dockerfile d.build_image "/docker_builds", args: "-t li-nginx-alpine -f /docker_builds/Dockerfile.li-nginx-alpine" # This is the same as # docker build -t li-nginx-alpine -f /docker_builds/Dockerfile.li-nginx-alpine /docker_builds/ end # Provisioner docker_compose: install docker_compose on the VM! # Require plugin `vagrant-docker-compose` config.vm.provision "docker_compose_plugin", type: "docker_compose" # config.ssh # config.winrm # config.vagrant end
#Do a loop (1..3).each do |i| config.vm.define "node-#{i}" do |node| node.vm.provision "shell", inline: "echo hello from node #{i}" end end
21.5. Plugin
vagrant-docker-compose- enables a new provisioner Docker Compose
vagrant plugin install vagrant-docker-composehttps://github.com/leighmcculloch/vagrant-docker-compose
Vagrant.configure("2") do |config| config.vm.box = "ubuntu/trusty64" config.vm.provision :docker config.vm.provision :docker_compose end
vagrant-vbguest- automatically installs VirtualBox Guest Additions (VBGA) on Vagrant VM
- Install
vagrant plugin install vagrant-vbguest- (no term)
- To prevent updating VBGA when
vagrant up, addconfig.vbguest.auto_update = falsein Vagrantfile. This happens when .box file has older version of vbguest while Vagrant on host has a newer version
vagrant-disksize- resize (upsize only) primary disk for Vagrant instance
vagrant plugin install vagrant-disksizeInsert this and run
vagrant haltandvagrant upvagrant.configure('2') do |config| config.disksize.size = '50GB' end
- While
vagrant up, something like this shows up, which means it is resized==> default: Resized disk: old 10240 MB, req 20480 MB, new 20480 MB
vagrant sshand check if disk size is increaseddf -h, if not, then "You may need to resize the filesystem from within the guest."- Comment out
config.disksize.size = '50GB'in Vagrantfile as it's unnecessary to upsize primary disk from now on
- Update all installed plugins
vagrant plugin update- List all installed plugins
vagrant plugin list- Uninstall a plugin
vagrant plugin uninstall vagrant-docker-compose- (no term)
- After Vagrant core is upgraded, you might need to auto upgrade existing plugins
vagrant plugin update vagrant-vbguestvagrant plugin expunge --reinstallvagrant plugin update
21.6. Box
A VirtualBox, .box file, is a compressed box on top of which a VM can run.
Box has a namespace username/boxname as in hashicorp/precise64
# export the running VM to a new box file vagrant package --output boxfile.box # decompress a box file to ~/.vagrant.d/boxes vagrant box add boxname boxfile.box # boxname is an installed Vagrant box # destroy current running Vagrant box vagrant destroy -f # and rm Vagrantfile # Setup another Vagrantfile which uses the newly created box cofnig.vm.box = "boxname" # To package an installed Vagrant box (of a specific provider and version) # into a .box file vagrant box repackage <boxname> <provider> <version> # remove a vagrant box vagrant box remove boxname # or vagrant box remove boxname --box-version=0.1.7 # List all installed Vagrant boxes vagrant box list # Check if the box used in the current folder is up-to-date vagrant box outdated # if it's not outdated and before destroy current Vagrant machine, update the box vagrant box update vagrant box update --box boxname # Destroy the current Vagrant machine and up again.
Vagrant .box should have these qualities:
- vagrant as a user and map /vagrant to the current folder in host
- be able to install as the Vagrant provision plugin instructs
- The ubuntu/xenial64 has a lot of trouble with that and hence it's not a good Vagrant box to start with
21.6.1. bento/ubuntu-16.04
vagrant box add bento/ubuntu-16.04
vagrant box add bento/ubuntu-18.04
21.7. Custom Box
21.8. SSH using Putty
- Download PuttyGen to accept OpenSSH key as vagrant provides
- PuttyGen > Load >
~/.vagrant.d/folder and select browse all files, select insecure_public_key - PuttyGen > Save public key file as insecure_public_key.ppk and save it to the same folder. Close PuttyGen and go back to Putty
- Putty > Connection > SSH > Auth, browse and select that ppk file.
- Connect using Putty!
- Specify the username for login: Putty > Connection > Data > Auto-login username
~/.vagrant.d/is the global. Different box has./.vagrant/folder. Runvagrant ssh-configto identify the $HOSTNAME $PORT $LOGINNAME
Convert the Identityfile to .ppk with SSH-2-RSA with 2048 bits and save it to the same folder.
22. Apache
22.1. Basics
Run this to get Apache version, root and config file httpd -V or apache2 -v in Ubuntu
22.2. Config loading
apt-get update; apt-get install apache2- Refer to image:php:apache to get configs
apache2ctl -Vorhttpd -Vto look forHTTPD_ROOTandSERVER_CONFIG_FILEto find the apache:config:main fileapache2ctl -SVirtualHost configuration: *:80 172.17.0.7 (/etc/apache2/sites-enabled/000-default.conf:1) # run settings followed ServerRoot: "/etc/apache2" Main DocumentRoot: "/var/www/html" Main ErrorLog: "/var/log/apache2/error.log" Mutex default: dir="/var/lock/apache2" mechanism=fcntl Mutex mpm-accept: using_defaults Mutex watchdog-callback: using_defaults Mutex rewrite-map: using_defaults PidFile: "/var/run/apache2/apache2.pid" Define: DUMP_VHOSTS Define: DUMP_RUN_CFG User: name="www-data" id=33 Group: name="www-data" id=33
22.2.1. Loaded Modules
- Show all loaded modules
apache2ctl -M- (no term)
- Sample loaded modules from image:php:apache
- (no term)
- You can also
ls -al /etc/apache2/mods-enabled
autoindex deflate expires filter mime rewrite setenvif mpm_prefork
22.2.2. sites-enabled VirtualHost
- Sites
- etc/apache2/sites-enabled
- (no term)
- Sample for image:php:apache
- (no term)
- Use Listen directive to listen ports other than the default 80 and 443. Refer to apache:ports.conf
<VirtualHost *:80> # The ServerName directive sets the request scheme, hostname and port that # the server uses to identify itself. This is used when creating # redirection URLs. In the context of virtual hosts, the ServerName # specifies what hostname must appear in the request's Host: header to # match this virtual host. For the default virtual host (this file) this # value is not decisive as it is used as a last resort host regardless. # However, you must set it for any further virtual host explicitly. #ServerName www.example.com ServerAdmin webmaster@localhost DocumentRoot /var/www/html # You may add Directory here # <Directory /var/www/html> # Options Indexes FollowSymLinks MultiViews # AllowOverride All # Order allow,deny # allow from all # </Directory> # Available loglevels: trace8, ..., trace1, debug, info, notice, warn, # error, crit, alert, emerg. # It is also possible to configure the loglevel for particular # modules, e.g. #LogLevel info ssl:warn ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined # For most configuration files from conf-available/, which are # enabled or disabled at a global level, it is possible to # include a line for only one particular virtual host. For example the # following line enables the CGI configuration for this host only # after it has been globally disabled with "a2disconf". #Include conf-available/serve-cgi-bin.conf </VirtualHost>
22.2.3. Main Config files
- Main config file (one of the following)
- /etc/apache2/apache2.conf
- /etc/httpd/httpd.conf
- /usr/local/apache/conf/httpd.conf
- Run apache:config to get Apache root directory to find config location
*.conffiles loading sequence# include module config IncludeOptional mods-enabled/*.load IncludeOptional mods-enabled/*.conf # include list of ports to listen on Include ports.conf # main config file config # generic IncludeOptional conf-enabled/*.conf # virutal host config IncludeOptional sites-enabled/*.conf
- Include https://github.com/h5bp/server-configs-apache/blob/master/dist/.htaccess inside <Directory> section
22.2.4. Restart
- Check config before restart. Wrong config (e.g. SSL) may prevent Apache from starting
apache2ctl configtestapachectl configtesthttpd -t
- Reload
sudo systemctl reload apache2.service- Ubuntu
/etc/init.d/apache2 reload- for apache that is initialized using command e.g. php apache docker container
apache2ctl graceful- to stop and start httpd daemon
sudo systemctl restart apache2apache2ctl restartservice apache2 restart/etc/init.d/apache2 restartservice httpd restart
- On Ubuntu, use
systemctl status apache2.service
22.2.5. a2ensite, a2dissite
/etc/apache2/sites-available/000-client1-site1.conf- Should have
<VirtualHost>block
- Should have
sudo a2ensite 000-client1-site1- create a symlink in
/etc/apache2/sites-enabled/
- create a symlink in
sudo a2dissite 000-client1-site1- remove a symlink in
/etc/apache2/sites-enabled
- remove a symlink in
- apache:reload
22.3. Modules
- Available modules
cat /etc/apache2/mods-available
- Enabled modules
cat /etc/apache2/mods-enabledapache2ctl -Mapache2ctl -M | grep headers_module
- Terms used to describe modules
- Status
- Base
- they're compiled and loaded into the server by default
- Status
Command line enable modules
a2enmod remoteip # some modules need to load config. If it doesn't specify, then the module does not need a2enconf a2enconf remoteip a2enmod rewrite
- 22.2.4
22.3.1. remoteip
https://github.com/wichon/docker-php-maxmind-geoip/blob/master/Dockerfile
When Nginx proxy Apache, Apache's remote_addr is the Nginx's IP (gateway of the current docker container)
Create remoteip.conf file under /etc/apache2/conf-available/remoteip.conf
RemoteIPHeader X-Forwarded-For RemoteIPProxiesHeader X-Forwarded-By
a2enmod remoteip && a2enconf remoteip # then apache:restart
22.3.2. mod_setenvif
<IfModule mod_setenvif.c> <IfModule mod_headers.c> <FilesMatch "\.(bmp|cur|gif|ico|jpe?g|png|svgz?|webp)$"> SetEnvIf Origin ":" IS_CORS Header set Access-Control-Allow-Origin "*" env=IS_CORS </FilesMatch> </IfModule> </IfModule>
22.3.3. mod_headers apache:mod_headers
- https://httpd.apache.org/docs/current/mod/mod_headers.html
- It provides directives to control and modify HTTP request and response headers. Headers can be merged, replaced or removed
<IfModule mod_headers.c> Header set Access-Control-Allow-Origin "*" </IfModule> <IfModule mod_headers.c> <FilesMatch "\.(eot|otf|tt[cf]|woff2?)$"> Header set Access-Control-Allow-Origin "*" </FilesMatch> </IfModule>
22.3.3.1. apache:mod_headers:cache
Refer to header:cache-control, 1.37
<filesMatch "\.(html|htm|js|css)$"> FileETag None <ifModule mod_headers.c> Header unset ETag Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate" Header set Pragma "no-cache" Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT" </ifModule> </filesMatch>
22.3.3.2. apache:mod_headers:header
- Syntax
Header [condition] add|append|echo|edit|edit*|merge|set|setifempty|unset|note header [[expr=]value [replacement] [early|env=[!]varname|expr=expression]]- (no term)
- All apache:directive context
- (no term)
- Eq. to nginx:d:add_header
- (no term)
- condition
always
- (no term)
- env
env=varname- applies if varname is defined
env=!varname- applies if varname is not defined
<IfModule mod_headers.c> Header unset X-Powered-By </IfModule>
Add flags for cookies
Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure
22.3.4. mod_include
22.3.5. mod_rewrite
22.3.5.1. RewriteEngine
- Syntax
RewriteEngine on|off- Default
RewriteEngine off- Context
- server config, virtual host, directory, .htaccess
- (no term)
- Usage
- In a particular context, disable all rewrite rules rather than commenting out all apache:rewriterule
- Each virtual host should have standalone
RewriteEngine onif the virtual hosts need to rewrite
<IfModule mod_rewrite.c> RewriteEngine on # ... </IfModule>
22.3.5.2. RewriteBase
Syntax: RewriteBase [URL-path] specifies the URL prefix to be used for per-directory (htaccess) RewriteRule directives that substitute a relative path
Request GET /somepath/localpath/pathinfo
RewriteBase "/somepath" RewriteRule ^localpath(.*) otherpath$1 # /somepath/otherpath/pathinfo RewriteRule ^localpath(.*) /otherpath$1 # /otherpath/pathinfo RewriteRule ^localpath(.*) http://thishost/otherpath$1 # /otherpath/pathinfo
22.3.5.3. RewriteCond
- Syntax
- RewriteCond [TestString] [CondPattern] [flags]
- (no term)
- CondPattern can be
- regex
- is, not a directory
- is, not a file
RewriteCond %{REMOTE_ADDR} ^1\.2\.3\.4$ RewriteCond %{HTTP_HOST} ^xyz\-40\.ca$ [OR] RewriteCond %{HTTP_HOST} ^www\.xyz\-40\.ca$ RewriteRule ^(.*)$ "https\:\/\/www\.xyz\.com\/$1" [R=302,L]
- Server-Variables
- https://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritecond
- https://abc.com:8080/path/to/file
- %{HTTP_HOST} %{SERVER_PORT} %{QUERY_STRING}
on/path/to/fileleading slash- %{REQUEST_FILENAME}
- The full local filesystem path if a file matches the request. Otherwise, REQUEST_URI
${HTTP_COOKIE}
RewriteCond %{HTTP_COOKIE} its=([^;]+) RewriteCond %1 ^me$ RewriteRule ...... RewriteCond %{HTTP_COOKIE} its=([^;]+) RewriteCond %{unescape:%1} ^me$ RewriteRule ......
%{TIME_*}
<IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^dealdays/?(.*)$ /DealDays/$1 [R=302,L] RewriteCond %{TIME_YEAR}%{TIME_MON}%{TIME_DAY}%{TIME_HOUR} >2017110521 RewriteRule ^DealDays/?(.*)$ / [R=302,L] </IfModule>
- RewriteCond Flags
- NC
- no case
- NV
- no vary
- (no term)
- OR
RewriteCond "%{REMOTE_HOST}" "^host1" [OR] RewriteCond "%{REMOTE_HOST}" "^host2" [OR] RewriteCond "%{REMOTE_HOST}" "^host3" RewriteRule ...some special stuff for any of these hosts...
22.3.5.4. RewriteRule
- Syntax
RewriteRule [Pattern] [Substitution] [flags]- Context
- server config, virtual host, directory, .htaccess
- Require
- apache:rewriteengine on
- (no term)
- Similar to 23.8
- (no term)
- Pattern
Match what?
- Virtual Host context
'/app1/index.html'
- The part of URL after the hostname and port and before the query string
- Decoded
- Per-directory context (Directory and .htaccess)
'app1/index.html'
- A relative path could be one of the following
- current directory's
.htaccess <Directory "/path/to/app/">)- apache:rewritebase
- Previous RewriteRule substition
- apache:documentroot or apache:alias
- current directory's
- regex
| ^ $ ( ) [ ] . * + ?\d- a-Z, 0-9 and underscore
- (no term)
- Substitution
"-"or-- the requested URI is not modified
- (no term)
- If it doesn't have a leading slash, the end URL has the current directory's .htaccess or apache:rewritebase
- (no term)
- Use double quotes
- Pattern and Substitution can be wrapped with double quotes
- When
#is used in Pattern or Substituion. If it is in Substition, 22.3.5.5 NE is required
22.3.5.5. RewriteRule Flags
- https://httpd.apache.org/docs/2.4/rewrite/flags.html
- Use comma to separate flags NOT spaces!
- return 403 and L is implied e.g.
RewriteRule "\.exe" "-" [F] - Last
- no case
- no escape. By default, special characters in Substituion wrapped in quotes will be escaped e.g.
& ? # - set MIME type with which the resulting response will be sent
- set environment variable e.g. [E=VAR:VAL] [E=VAR] (set to empty) [E=!VAR] (unset a previously set env var named VAR)
- handle request by mod_proxy
- Default behavior of RewriteRule is to discard the existing query string, and replace it with the newly generated one. Use it to cause the query string to be combined
- e.g.
RewriteRule "/pages/(.+)" "/page.php?page=$1" [QSA]/pages/123?one=twowill be mapped to/page.php?page=123&one=two
- There's no QSA equivalent in Nginx because by default Nginx appends query parameters
- e.g.
- force an external redirect, optionally with the specified HTTP status code
22.3.5.6. Rewrite a path to sub directory with .htaccess
- URL
/inventory/*rewrite to/my-app/public/*./my-app/public/index.phpis where the app starts./my-app/app/*.*are the app codes./my-app/public/theme/*.*and./my-app/public/uploads/*.*have the assets
./.htaccess
<IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^inventory/?$ /my-app/public/index.php [L] RewriteRule ^inventory/(.*)$ /my-app/public/$1 [L] </IfModule>
./my-app/public/.htaccess
<IfModule mod_rewrite.c> RewriteEngine On RewriteBase /my-app/public/ RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /my-app/public/index.php [L] </IfModule>
22.3.5.7. WordPress Example
- RewriteRule with redirect flag [R] is to redirect URL. Without it, RedirectRule maintains the URL on LHS in the address bar.
Redirect and RedirectMatch is to redirect Redirect means the URL in the address bar will change
RewriteRule without proxy flag [P] is to maintain the URL on LHS as $_SERVER['REQUEST_URI'] in RewriteRule RewriteRule with [P] is to change the URL to the one on RHS as the new $_SERVER['REQUEST_URI'] in RewriteRule and pass it to the proxy [P] requires mod_proxy enabled
The only way to modify $_SERVER['REQUEST_URI'] is to do a redirect or with the [P] flag.
RewriteRule is relative to the directory that the current .htaccess is in for both RHS and LHS. RewriteBase only affects the dest of RewriteRule (RHS) only if dest is a relative path
Redirect /dealdays to /DealDays
# custom RewriteRule goes before the wordpress block <IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^dealdays/?(.*)$ /DealDays/$1 [R=302,L] RewriteRule ^abc/?(.*)$ /subdir/abc.php [L] </IfModule> # custom RewriteRule end # BEGIN WordPress <IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L] </IfModule> # END WordPress # default wordpress htaccess block is to redirect to index.php if it's not a file and not a directory # Rules below the block has to be either a file or an existing directory RewriteRule ^dealdays/(.*)$ /DealDays/$1 [R=302,L] # If redirect to another url, use # Redirect is prefix match and append additional path info beyond the matched URL-path to the new/target URL. # a.com/dealdays2 to b.com/dealdays2, a.com/dealdays2/foo.txt to b.com/dealdays2/foo.txt, a.com/dealdays2/xyz to b.com/dealdays2/xyz # Redirect syntax :: Redirect [status] [URL-path] URL # [URL-path] is case-sensitive beginning with slash. relative path is not allowed # URL may be either an absolute URL beginning with scheme and hostname or a URL-path beginning with a slash Redirect 301 /dealdays2 http://b.com/dealdays2 # Redirect without appending additional path info # RedirectMatch is like Redirect but regex # syntax :: RedirectMatch [status] regex URL RedirectMatch 302 ^/dealdays2/?$ http://b.com/dealdays2 # case insensitive just add (?i) in front of the pattern RedirectMatch 302 (?i)^dealdays2/?$ /dealdays/
Redirect root domain only but not other URI. e.g. mysite.com or www.mysite.com redirect to abc.com but not mysite.com/xyz to abc.com/xyz
RewriteEngine on RewriteCond %{HTTP_HOST} ^(www\.)?mysite\.com [NC] # no www :: RewriteCond %{HTTP_HOST} ^mysite\.com [NC] RewriteCond %{REQUEST_URI} ^/$ # Rewriterule ^(.*)$ http://abc.com/ [L,R=302] Rewriterule ^$ http://abc.com/ [L,R=302]
Redirect www.abc.com to abc.com
RewriteCond %{HTTP_HOST} ^www.abc.com [NC] RewriteRule ^(.*) http://abc.com/$1 [L,R=302]
22.3.6. mod_autoindex
If the URI matches a directory but doesn't specify an index file e.g. index.html, by default Apache returns the index of that directory. Don't do that!
<IfModule mod_autoindex.c> Options -Indexes </IfModule>
22.3.7. mod_mime
22.3.7.1. AddType TypesConfig apache:mod_mime:addtype
- Context: all
AddType media-type extension [extension] ... AddType image/webp .webp AddType font/woff2 woff2 AddType image/jpeg jpeg jpg jpe
/etc/apache2/mods-available/mime.conf loads MIME types using TypesConfig /etc/mime.types
echo "font/woff2 woff2" >> /etc/mime.types
22.3.7.2. AddEncoding
AddEncoding encoding extension [extension] … extension may or may not have leading dot. This doesn't encode the file but rather sets the response header Content-Type. See apache:mod_filter:addoutputfilterbytype
<IfModule mod_mime.c> AddEncoding gzip svgz </IfModule>
22.3.8. mod_dir
22.3.8.1. DirectoryIndex
DirectoryIndex index.html index.txt /cgi-bin/index.pl # http://example.com/docs/index.html # could cause the CGI script /cgi-bin/index.pl to be executed if neither index.html or index.txt existed in a directory.
22.3.9. mod_filter
22.3.9.1. AddOutputFilterByType apache:mod_filter:addoutputfilterbytype
AddOutputFilterByType filter[;filter…] media-type [media-type] … Default deflate content types :: /etc/apache2/mods-available/deflate.conf
<IfModule mod_filter.c>
AddOutputFilterByType DEFLATE "application/atom+xml" \
"application/javascript" \
"application/json" \
...
"image/svg+xml" \
...etc.
</IfModule>
22.3.10. mod_alias
22.3.10.1. Redirect apache:mode_alias:Redirect
22.4. Directive Context apache:directive context
- server config
- This means that the directive may be used in the server configuration files (e.g., httpd.conf), but not within any <VirtualHost> or <Directory> containers
- Not allowed in .htaccess files at all.
- virtual host
- This context means that the directive may appear inside <VirtualHost> containers in the server configuration files
- directory
- A directive marked as being valid in this context may be used inside <Directory>, <Location>, <Files>, <If>, and <Proxy> containers in the server configuration files, subject to the restrictions outlined in Configuration Sections
.htaccess- If a directive is valid in this context, it means that it can appear inside per-directory .htaccess files. It may not be processed, though depending upon the overrides currently active
22.5. Core Directives
These can be used in all apache:directive context. However, some directive setting needs to happen in a different context. https://httpd.apache.org/docs/2.4/mod/core.html
ErrorDocument 404 /404.html # This setting prevents Apache from returning a 404 error as the result # of a rewrite when the directory with the same name does not exist. Options -MultiViews
22.5.1. AllowOverride
AllowOverride All|None|directive-type [directive-type] … context: only directory default: AllowOverride None (2.3.9 and later), AllowOverride All (2.3.8 and earlier)
Which directives declared in .htaccess can override earlier configuration directives.
None :: .htaccess files are completely ignored.
22.5.2. FilesMatch
Force to download a PDF file. The path doesn't matter
<FilesMatch "aPDFfile.pdf"> ForceType application/octet-stream Header add Content-Disposition "attachment" </filesMatch>
22.5.3. ServerSignature
ServerSignature On|Off|EMail- off
- server config, virtual host, directory, .htaccess
22.5.4. LogLevel
- Syntax
LogLevel [module:]level [module:level] ...- Default
LogLevel warn- Context
- server config, virtual host, directory
- (no term)
- https://httpd.apache.org/docs/2.4/mod/core.html#loglevel
- (no term)
- Put it inside VirtualHost e.g. in /etc/apache2/sites-available/*.conf
# for all modules use alert level, but for rewrite module, level is trace6, which is extremely detailed LogLevel alert rewrite:trace6 # sometimes, mod_rewrite.c:trace6 is required # For < 2.4 use this for Rewrite log RewriteEngine On RewriteLog "/var/log/apache2/rewrite.log" RewriteLogLevel 3
trace1 to trace8 (most detailed)
22.5.5. Include
- Syntax
Include file-path|directory-path|wildcard- Context
- server config, virtual host, directory
- Relative path
- relative to ServerRoot e.g.
/etc/apache2
Include conf/ssl.conf # /etc/apache2/conf/ssl.conf Include conf/vhosts/*.conf Include /path/to/conf/*.conf # fail with an error if there is no file matched
22.6. LogFormat
- LogFormat
- https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#logformat
LogFormatassociate a format string to a nickname (e.g. common), nickname is optionalCustomLogspecifies the log file location for that nickname
- Context
- server config, virtual host
- Default
LogFormat "%h %l %u %t \"%r\" %>s %b"- Custom Log Formats
- https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#formats
LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog logs/access_log common
Combined more fields
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined CustomLog log/access_log combined # 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 # after combined # 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"
2326 :: %b :: size of the object returned to the client, not including the response headers. If no content was returned to the client, this value will be "-". To log "0" for no content, use %B instead.
22.7. Basic authentication
htpasswd creates auth file and it needs this package. htpasswd doesn't need to be installed on server.
apt-get install apache2-utils # use -c to create and sammy is the username, supply password later sudo htpasswd -c /mywebsite/.htpasswd-dev sammy # later add more user without -c sudo htpasswd -c /mywebsite/.htpasswd-dev another_user
.htpasswd file content
Output sammy:$apr1$.0CAabqX$rb8lueIORA/p8UzGPYtGs/ another_user:$apr1$fqH7UG8a$SrUxurp/Atfq6j7GL/VEC1
MD5 \(apr1\).0CAabqX$rb8lueIORA/p8UzGPYtGs/ 32bit salt is .0CAabqX MD5 result is rb8lueIORA/p8UzGPYtGs/
22.7.1. Option 1: control access within Virutal Host Definition
/etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# add this
<Directory "/var/www/html">
AuthType Basic
AuthName "Restricted Content"
AuthUserFile /etc/apache2/.htpasswd
Require valid-user
</Directory>
# add this.
</VirtualHost>
22.7.2. Option 2: control access with .htaccess
Main config file :: etc/apache2/apache2.conf
Allow .htaccess to override config
For /var/www, change AllowOverride None to All
. . . <Directory /var/www/> Options Indexes FollowSymLinks AllowOverride All Require all granted </Directory> . . .
var/www/mywebsite.htaccess, restart apache
AuthType Basic AuthName "restricted area" AuthUserFile /var/www/__secure/.htpasswd-dev require valid-user
22.8. Production Setup apache:production
This guide is for httpd. Optimize Webhost Setup
nano /etc/httpd/conf/httpd.conf
Timeout 30 KeepAlive on MaxKeepAliveRequests 50 KeepAliveTimeout 15 #Changes to prefork module <IfModule prefork.c> StartServers 3 MinSpareServers 2 MaxSpareServers 5 ServerLimit 256 MaxClients 10 MaxRequestsPerChild 1000 </IfModule>
Search AllowOverrride and set it to All for
<Directory /> Options FollowSymLinks AllowOverride All </Directory>
And
# AllowOverride controls what directives may be placed in .htaccess files. # It can be "All", "None", or any combination of the keywords: # Options FileInfo AuthConfig Limit # AllowOverride All
Restart Apache service httpd restart
22.9. Force redirect from non www to www
<IfModule mod_rewrite.c> RewriteEngine On RewriteCond %{HTTP_HOST} !^www\. [NC] RewriteRule ^(.*)$ http://www.%{HTTP_HOST}%{REQUEST_URI} [R=301,L] </IfModule>
Redirect from www.example.com to example.com
<IfModule mod_rewrite.c> RewriteEngine On RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC] RewriteRule ^ %{ENV:PROTO}://%1%{REQUEST_URI} [R=301,L] </IfModule>
22.10. Force redirect to https .htaccess
Diffrent hositng companies have different rule
In general
<IfModule mod_rewrite.c>
RewriteEngine On
# RewriteBase is optional but some hosting requires it
RewriteBase /
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
</IfModule>
InMotion https://www.inmotionhosting.com/support/website/ssl/how-to-force-https-using-the-htaccess-file
<IfModule mod_rewrite.c> RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$ RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$ RewriteEngine On RewriteCond %{SERVER_PORT} 80 RewriteRule ^(.*)$ https://www.example.com/$1 [R=301,L] </IfModule>
22.11. Restrict access to a file or sub folder
var/www.htaccess blocks access to folders cs-devops and .git and file .gitignore. This way log gets 403 and apache doesn't need to be restarted
RewriteRule ^(cs-devops/|\.git/|\.gitignore|docker-compose\.yml|Makefile|README\.md|error_log|wp-config-li-[^.]+\.php) - [F,NC]
22.12. WebP
<IfModule mod_rewrite.c>
RewriteEngine On
# Check if browser support WebP images
RewriteCond %{HTTP_ACCEPT} image/webp
# Check if WebP replacement image exists
RewriteCond %{DOCUMENT_ROOT}/$1.webp -f
# Serve WebP image instead
RewriteRule (.+)\.(jpe?g|png)$ $1.webp [T=image/webp,E=accept:1]
</IfModule>
<IfModule mod_headers.c>
Header append Vary Accept env=REDIRECT_accept
</IfModule>
AddType image/webp .webp
22.13. SSL apache:ssl
- Docs from DigiCert
- Find config file that handles SSL
grep -i -r "SSLCertificateFile" /etc/httpd/orgrep -i -r "SSLCertificateFile" /etc/apache2/
sudo a2enmod ssl- The
SSLCertificateFilemay also contain intermediate certificates, DH parameters and EC curve. Add
SSL*directives# May need for port 80 as well <VirtualHost 192.168.0.1:80> DocumentRoot /var/www/html2 ServerName www.yourdomain.com SSLEngine on SSLCertificateFile /path/to/your_domain_name.crt SSLCertificateKeyFile /path/to/your_private.key SSLCertificateChainFile /path/to/DigiCertCA.crt </VirtualHost> <VirtualHost 192.168.0.1:443> DocumentRoot /var/www/html2 ServerName www.yourdomain.com SSLEngine on SSLCertificateFile /path/to/your_domain_name.crt SSLCertificateKeyFile /path/to/your_private.key SSLCertificateChainFile /path/to/DigiCertCA.crt </VirtualHost>
- Check before apache:restart
22.14. Redirect to HTTPS apache:https
RewriteEngine On RewriteCond %{SERVER_PORT} 80 # or # RewriteCond %{HTTPS} off # test one page # RewriteCond %{REQUEST_URI} ^/page-path-name/$ RewriteRule ^(.*)$ https://www.example.com/$1 [R,L] # or # RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R,L]
A specific domain
RewriteEngine On RewriteCond %{HTTP_HOST} ^example\.com [NC] RewriteCond %{SERVER_PORT} 80 RewriteRule ^(.*)$ https://www.example.com/$1 [R,L]
Force SSL on a specific folder. Place this .htaccess in that folder
RewriteEngine On RewriteCond %{SERVER_PORT} 80 RewriteCond %{REQUEST_URI} folder RewriteRule ^(.*)$ https://www.example.com/folder/$1 [R,L]
23. Nginx
23.1. Install on Ubuntu 16.04
# Ubuntu 16.04 sudo apt-get update sudo apt-get install nginx
23.1.1. Config Firewall nginx:ufw
This is optional if firewall ufw doesn't exist.
Nginx registers itself as a service with firewall ufw
sudo ufw app list # Available applications: # Nginx Full # Nginx HTTP # Nginx HTTPS # OpenSSH
- Nginx Full
- opens both port 80 (normal, unencrypted web traffic) and port 443 (TLS/SSL encrypted traffic)
- Nginx HTTP
- opens only port 80 (normal, unencrypted web traffic)
- Nginx HTTPS
- opens only port 443 (TLS/SSL encrypted traffic)
Enable Nginx HTTP
sudo ufw allow 'Nginx HTTP' sudo ufw status # output # Status: active # To Action From # -- ------ ---- # OpenSSH ALLOW Anywhere # Nginx HTTP ALLOW Anywhere # OpenSSH (v6) ALLOW Anywhere (v6) # Nginx HTTP (v6) ALLOW Anywhere (v6)
Enable HTTPS, allow Full and delete HTTP
sudo ufw allow 'Nginx Full' sudo ufw delete allow 'Nginx HTTP'
23.1.2. Service Check, reload, restart, Check syntax
- Check syntax for config before restart ::
sudo nginx -t - Nginx can be run with daemond off by
nginx -g 'daemon off;'- Then use
/etc/init.d/nginxinstead ofsystemctlnorservice
- Then use
systemctl stop/start/restart/reload/disable/enable nginx- status
systemctl status nginx- restart
service nginx restart- or
sudo /etc/init.d/nginx restart
- or
- (no term)
- reload config files only
sudo nginx -s reload- or
sudo /etc/init.d/nginx reload restartdrops connection, usereloadto not drop connection
- disable
- nginx service, default the nginx service is to start automatically
nginx -vOutput● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2016-04-18 16:14:00 EDT; 4min 2s ago Main PID: 12857 (nginx) CGroup: /system.slice/nginx.service ├─12857 nginx: master process /usr/sbin/nginx -g daemon on; master_process on └─12858 nginx: worker process
23.2. /etc/nginx /var/log/nginx
23.2.1. etc/nginx/sites-available
The directory where per-site "server blocks" can be stored.
Nginx will not use config files in this directory unless they are linked to the sites-enabled directory.
There should be a default file you can use. /etc/nginx/sites-available/default
server { listen 80 default_server; listen [::]:80 default_server; root /var/www/html; index index.html index.htm index.nginx-debian.html; server_name _; location / { try_files $uri $uri/ =404; } }
ATTENTION! Only one server block can have default_server Search if there is any file has default_server
grep -R default_server /etc/nginx/sites-enabled/
sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/example.com sudo nano /etc/nginx/sites-available/example.com
/etc/nginx/sites-available/example.com
server {
listen 80;
listen [::]:80;
root /var/www/example.com/html;
index index.html index.htm index.nginx-debian.html;
server_name example.com www.example.com;
location / {
try_files $uri $uri/ =404;
}
}
Enable the server block and restart Nginx
sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com
In order to avoid a possible hash bucket memory problem that can arise from adding additional server names, we will go ahead and adjust a single value within our /etc/nginx/nginx.conf file. Open the file now:
sudo nano /etc/nginx/nginx.conf
Within the file, find the server_names_hash_bucket_size directive. Remove the # symbol to uncomment the line:
http {
. . .
server_names_hash_bucket_size 64;
. . .
}
Check syntax
sudo nginx -t sudo systemctl restart nginx
23.2.2. etc/nginx/sites-enabled
The directory where enabled per-site "server blocks" are stored.
Niginx on Debian or Ubuntu include /etc/nginx/sites-enabled/* while other installation have server blocks set up /etc/nginx/conf.d/*.conf
23.2.3. etc/nginx/snippets
This directory contains config fragments that can be included elsewhere in the Nginx config.
23.2.4. /var/log/nginx/access.log
- Every request to your web server is recorded
- Refer to nginx:d:log_format
23.2.5. /var/log/nginx/error.log
- Refer to nginx:d:error_log
23.3. Installed modules
# To check if a module (http_stub_status_module) is installed. It returns the module name if it's installed nginx -V 2>&1 | grep -o with-http_stub_status_module
23.3.1. Stub status module
- https://nginx.org/en/docs/http/ngx_http_stub_status_module.html
Check if stub status module is installed, if it returns
with-http_stub_status_modulethen it's installednginx -V 2>&1 | grep -o with-http_stub_status_module
Create this location directive under any server directive to see the status info for the whole Nginx status for all servers and connections
server { listen 81; server_name localhost; access_log off; allow 127.0.0.1; deny all; location /nginx_status { # Choose your status module # freely available with open source NGINX stub_status; # for open source NGINX < version 1.7.5 # stub_status on; # available only with NGINX Plus # status; # ensures the version information can be retrieved server_tokens on; } }
23.4. Config
- Check syntax and restart nginx:restart
Check a setting
cd /etc/nginx/ grep -r 'client_max_body_size' .
23.4.1. Main global config /etc/nginx/nginx.conf
user www-data; worker_processes 4; pid /run/nginx.pid; events { ... } http { sendfile off; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; access_log /var/log/nginx/access.log ; error_log /var/log/nginx/error.log; include /etc/nginx/conf.d/*.conf; }
23.4.2. Define server(s) in /etc/nginx/conf.d/default.conf
# comment starts here. # You should always add a server to catch any requests without "Host" header server { listen 80; server_name ""; # this line can be omitted if version > 0.8.48 return 444; } # another server server { }
23.5. server Block nginx:b:server
- Include directives
- listen, server_name, root, index
- Include blocks
- location
server { listen 80 default_server; # or without any default_server in listen, the first server with listen is the default server # listen 80; # listen actually is a combo of ip and port 192.168.1.1:80 # nginx first identify which ip:port has the incoming request, then try to match the server_name from the request # if no server_name matches, then it goes to the default_server for ip:port # If no ip is set in ip:port, then 0.0.0.0 is used # If no port is set in ip:port, then 80 is used # If no listen is set, then 0.0.0.0:80 is used # If only ip is specified (no port), it has a higher specificity than only port is specified. # 0.0.0.0 means all ipv4 addresses # listen [::]:80 ipv6only=on; # means to listen all ipv6 address :::80 server_name example.org www.example.org; # server_name :: can contain wildcards that either at start or end. # *.example.org matches www.example.org and www.sub.exmple.org as well. # nginx first searches for the exact match of a prefix given by literal strings (no regex, no wildcard) # If not found, then checks for leading wildcard first then trailing wildcard (no regex) # In each case if there're multiple matches, the longest match will be used # Then nginx checks server_name given by regex in the order listed # The first matching regex stops the search and nginx will use that server_name # If no regex is matched, then uses the most specific prefix found earlier # server_name regex always starts with ~. Recommended to always start with ^ and end with $ # e.g. server_name ~^www\d+\.example\.net$; # regex containing { and/or } should be quoted entirely # e.g. server_name "~^(?<name>\w\d{1,3}+)\.example\.net$" # named capture :: later $name can be used # always use named capture as $1 will be passed down to any sub directives and server_name is usually # one of the top directives # Special server_name :: "" you can do this # server_name example.org ""; # This matches any server name # server_name _; index index.cfm index.html index.htm; root /var/www; # root combines with uri to get a file. e.g. /var/www/uripart1/uripart2 # If the result is a directory and doesn't have an index file (e.g. index.cfm set in index) # Then it will return 403 forbidden # Turn on autoindex will list files in directory # autoindex on; # include any common config files. /etc/nginx/common/*.conf include common/common.conf }
23.6. location block
23.6.1. Basics
- location blocks are usually under server blocks
- https://nginx.viraptor.info/
- Don't need to escape
/ (and)
# location syntax # location optional_modifier location_match {...} # location only matches URI without URL parameters # modifier = location = /page1 { # exact match # if there is a match, return this immediately } # modifier ^~, non regex match location ^~ /site { # This is a prefix match: /site/* # If this match (with ^~) is the longest prefix match, return immediately } # no modifier, the location_match part is a prefix location /site { # This is also a prefix match: handles /site, /site/*, /site/page1/index.html # If this match (without ^~) is the longest prefix match, # save for the moment and move on to regex matches } # modifier ~ or ~*, regex match location ~ \.(jpe?g|png|gif|ico)$ { # ~ :: case-sensitive, # ~* :: is case-insensitve # By now, if there is a prefix match (also longest prefix match), it's already stored in memory # These regex matches are checked by the listed order # Regex match searching terminates on the first match, and return immediately # If no regex match, then return the previous found longest prefix match # So if there's a regex match, then the regex match will be used # Also notice, if there're any regex matches that are within the longest prefix match, # those will be evaluated, in listed order, before any of the other regex matches. # Regex match inside prefix match > Stand alone regex match even though there's prefix match } location ~ ^/(digital-archive|digital-edition)/?$ { # $ can be used to mark the end return 302 http://www.target.ca/pub/abc/; } location ~ ^/(subscribe|contacts|advertise|privacy-policy)/?$ { # matched: /subscribe /subscribe/ # not matched: /subscribe/abc rewrite (?i)^/([^/]+)$ https://www.target.ca/$1 last; } location ~ ^/(segment|keyword|category|tag)/[^/]+ { # matched: /tag/abc # not matched: /tag/ /tag rewrite (?i)/(.*)$ https://www.target.ca/$1 last; } location ~ ^/[^/]+/?$ { # matches https://source.ca/abc and /abc/ # but not https://source.ca/abc/xyz } # one location can "jump" to another location root /var/www/main; location / { rewrite ^/rewriteme/(.*)$ /$1 last; try_files $uri $uri.html $uri/ /fallback/index.html; } location /fallback { root /var/www/another; } # /rewriteme/hello > /hello > /hello, /hello.html, /hello/, /fallback/index.html # one location can "jump" to another location :: custom 404 root /var/www/main; location / { error_page 404 /another/whoops.html; } location /another { root /var/www; }
23.6.2. Subdirectory as a React app
server { gzip on; gzip_types text/css application/x-javascript application/javascript text/javascript text/x-javascript; gzip_vary on; listen 80; access_log /var/log/nginx/access.log gslog if=$loggable; server_name localhost; root /var/www/; client_max_body_size 120M; location / { index index.php; autoindex off; } location /path/to/react/app { index index.php; autoindex off; try_files $uri $uri/ /path/to/react/app/index.php; } location ~ \.php$ { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_keep_conn on; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_send_timeout 3600; fastcgi_read_timeout 3600; fastcgi_connect_timeout 3600; send_timeout 3600; include fastcgi_params; } }
23.7. proxy_pass directive
23.7.1. Basic
If proxy_pass is specified with a URI, then when a request is passed to the server, the part of a normalized request URI matching the location is replaced by a URI specified in the directive
location /name/ {
proxy_pass http://127.0.0.1/remote/;
# or http://127.0.0.1/
}
If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI
location /some/path/ {
proxy_pass http://127.0.0.1;
}
proxy_pass can have URI parts if it's under a non-regex location block
location / {
proxy_pass http://127.0.0.1:8888/long/uri;
# There's no trailing '/'
}
proxy_pass without URI under a regex location block will pass URI to the proxy
location ~* \.(cfm|...|html)$) {
proxy_pass http://127.0.0.1:8888;
}
proxy_pass with URI under a regex location block will throw an error. To avoid error, set a variable which holds the original or modified URI to pass
location ~* \.(cfm|...|html)$) {
set $request_url /site2$request_uri;
if ($request_uri ~* ^/legacy_js) {
set $request_url $request_uri;
}
proxy_pass http://127.0.0.1:8888$request_url;
}
23.7.2. Sample
location / { proxy_pass http://127.0.0.6:8888; proxy_set_header Host 127.0.0.6; proxy_redirect off; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; }
Common docker container nginx:proxy:docker
server { listen 80; server_name dev.myweb.com; # refer to SSL config for nginx:ssl:server block for Let's Encrypt # deny .git location ~ /\.git { deny all; } location / { proxy_pass http://127.0.0.1:9977; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $http_host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # proxy_set_header X-Forwarded-SSL on; # for ssl # proxy_redirect default; # default is to redirect location into proxy_pass # proxy_redirect changes the response header ~Location~ from the proxied server } }
$http_host equals always the HTTP_HOST request header. $host equals $http_host, lowercase and without the port number (if present), except when HTTP_HOST is absent or is an empty value.
- In that case, $host equals the value of the server_name directive of the server which processed the request.
Refer to wordpress:ssl:behind proxy
proxy headers in PHP are HTTP_X_REAL_IP
23.7.3. Sample: Redirect a path to a different proxy nginx:redirect path to proxy
server { listen 80; server_name abc.com www.abc.com; # this doens't work in wordpress! => location ^~ /subpath location ^~ /subpath/ { proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $http_host; proxy_pass http://127.0.0.1:8787/; } location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $http_host; proxy_pass http://127.0.0.1:9977; } }
23.8. Rewrite directive
- http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite
rewrite regex replacement [flag];- —
- server, location, if
- Escape
- Don't need to escape forward slash
/ - See 8.6.9.2.1
- Don't need to escape forward slash
- https://regex101.com/r/xEWBb8/1/
- Regex type
- PCRE (PHP)
- Delimiter
- backtick `
- flags
- g i m (global, insensitive, multiline)
e.g. Source folder is CMS, and any case variants of CMS can be redirected
rewrite (?i)^/cms/?$ /CMS/index.cfm last; rewrite (?i)^/cms/(.*)$ /CMS/$1 last; # http://thissite.ca/ijk/xyz-123 => http://othersite.ca/citb/xyz rewrite (?i)/([^/]+)-(\d+)/?$ https://othersite.ca/citb/$1;
- Flags
last- Not to pass any more rewrite-module directives in the current
serverorlocationblock redirect- returns a temporary redirect with the 302 code; used if a replacement string does not start with
http://,https://, or$scheme permanent- same as redirect but 301
23.9. listen Directive
Default listen *:80; if Nginx is run as superuser or listen *:8000 otherwise.
Parameter ipv6only can be set only once across all config files. Default is on. Which means when [::] is used, listen only for IP v6.
listen [::]:443 ssl ipv6only=on;
23.11. add_header
- Context
- http, server, location, if in location
- Syntax
add_header name value [always]
Add to a response header. The value can contain variables.
- always
- By default the directive only adds header if the response code is in a certain range.
alwaysadds it regardless of the response code
add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always add_header X-Frame-Options "SAMEORIGIN" always; add_header X-XSS-Protection "1; mode=block" always;
23.12. Server name redirect
Use nginx:d:rewrite for regex redirect
server { listen 80; listen [::]:80; server_name .mydomain.com; # this matches *.mydomain.com return 301 http://www.adifferentdomain.com$request_uri; } # if you have ssl setup, you need to specify SSL info # refer to nginx:ssl:example server { # your original setting server_name .yoursite.com listen [::]:443 ssl ipv6only=on; # You might need to remove ipv6only=on; if it was set. managed by Certbot listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/yoursite.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/yoursite.com/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot return 301 https://adifferentdoamin.com$request_uri; }
23.13. error_log nginx:d:error_log
- Default
error_log logs/error.log error;- Syntax
error_log file [level];- Context
- main, http, mail, stream, server, location
- Level
- one of debug, info, notice, warn, error, crit, alert, or emerg
23.14. Core Module Directives
- client_max_body_size
- Maximum allowed size of the client request body. If the size in a request exceeds the configured value, the 413 (Request Entity Too Large) error is returned to the client. Please be aware that browsers cannot correctly display this error. Setting size to 0 disables checking of client request body size.
- Default is 1m
- Need to also change PHP
post_max_sizeas well. Restart PHP and then Nginx- The following PHP configs may also need to be changed
memory_limitupload_max_filesize
- The following PHP configs may also need to be changed
- https://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/
23.15. log_format and access_log nginx:d:log_format
log_format
- Default
log_format combined "...too long";- Syntax
log_format name [escape=default|json|none] string ...;- Context
- http
- (no term)
- Format variables
$msec- time in seconds with a milliseconds resolution at the time of the log write
$request_time- request processing time in seconds with a milliseconds resolution; time elapsed between the first bytes were read from the client and the log write after the last bytes were sent to the client
$upstream_connect_time- time spent establishing a connection with an upstream server
$upstream_header_time- time between establishing a connection to an upstream server and receiving the first byte of the response header
$upstream_response_time- tiem between establishing a connection to an upstream server and receiving the last byte of the response body
$status- response status
Format combined has this
log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';
access_log
- Default
access_log logs/access.log combined;- Syntax
access_log path [format [buffer=size] [gzip[=level]] [flush=time] [if=condition]];- Or
access_log off;format is not specified- then combined is used
- Context
- http, server, location, if in location, limit_except
23.15.1. Use Linux commands to analyze
cd /var/log/nginx # Filter archived logs with string `entity=loans&action=list`, grab the 17th column, sort and count the unique values zcat access.log.*.gz | grep 'entity=loans&action=list' | awk '{print $17}' | sort | uniq -c | sort -rn # cut -d '"' -f3 :: delimit the line with double quote and choose the 3rd element cat access.log | cut -d '"' -f3 | cut -d ' ' -f2 | sort | uniq -c | sort -rn
23.16. Let's Encrypt SSL nginx:ssl letsencrypt:certbot
- On Ubuntu, install Let's Encrypt SSL client
certbotpackage to obtain, install and renew certificate - means to modify webserver config e.g. Nginx
/etc/nginx/sites-available/mywebsite.conf - means to get and store certificates to
/etc/letsencryptdirectory- all live certificates
/etc/letsencrypt/live/example.com/fullchain.pem
- used to be
letsencryptin older version orcertbot-autoin an alternate installation method- Syntax
certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ... Commands
run- obtain and install
certonly- obtain or renew a certificate
- (no term)
renew
- Dry run to renew all certificates
certbot renew --dry-runcertificates
- Check all certificates status
certbot certificates
Options
--webroot -w MYWEBROOT_PATH- letsencrypt:certbot:webroot
-n- non-interactive
--agree-tos- agree to ACME server's Subscriber Agreement
--force-renewal,--renew-by-default- If a cert exists, renew it regardless of whether it's near expiry
--dry-run- dry run
- (no term)
- A plugin can be an authenticator and/or an installer. Use options to specify which plugin to use
-a,--authenticator-i,--installerDifferent plugins can be combinedcertbot run -a webroot -i apache -w /var/www/html -d example.com
- Syntax
/etc/letsencrypt/renewal/mydomain.com.conf/etc/letsencrypt/cli.ini(usually empty)- /var/log/letsencrypt
- it runs
certbot -q renewso renewal config file is very important for each domain
23.16.1. Challenges
- http challenge (port 80)
- requires to place a file with specific name and content in
your-website-root/.well-known/acme-challenge/directoy - dns (port 53 open on DNS server)
- requires to place a TXT DNS record with specific contents
- e.g. _acme-challenge.example.com. 300 IN TXT "gfj9Xq…Rg85nM"
- tls-sni (port 443)
- prepares a self-signed SSL certificate with the challenge validation encoded into a subjectAlternatNames entry. Config your SSL server to present this challenge SSL certificate to the ACME server using SNI.
- nginx and apache plugins use this type
23.16.2. certbot nginx apache plugins
They are authenticator and install plugins. tls-sni-01 challenge
sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install python-certbot-nginx # sudo apt-get install python-certbot-apache
- Allow HTTPS in firewall nginx:ufw
Obtain an SSL certificate using
--nginxplugin for those -d server namessudo certbot --nginx -d example.com -d www.example.com
certbotlooks for server block files with server_name- The first time running certbot, prompt to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the Let's Encrypt server, then run a challenge to verify that you control the domain you're requesting a certificate for
- After that, certificate is downloaded, Nginx config is updated and reloaded
/etc/nginx/sites-enabled/yoursite.com nginx:ssl:server block
server { listen 80; listen [::]:80; server_name example.com www.example.com; return 301 https://example.com$request_uri; # force redirect to https } server { # your original setting # listen 80; # if you want to serve http as well in the same block # ssl on; # is not needed when ssl is in listen directive listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/yoursite.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/yoursite.com/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot # follow by these proxy rules nginx:proxy:docker } # certbot will insert this block, but it's better to use the one atop. server { if ($host = yoursite.com) { return 301 https://$host$request_uri; } # managed by Certbot listen 80; server_name yoursite.com; return 404; # managed by Certbot }
- /etc/letsencrypt/live/example.com/fullchain.pem
- /etc/letsencrypt
- ssl:test
- Let's Encrypt SSL certificates are valid for 90 days. On Ubuntu, install
certbotpackage to renew certificate.certbot renewis run twice a day on systemd timer. For non-systemd,/etc/cron.dalso runs twice a day. /etc/cron.d/certbot
SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew
- Dry run
sudo certbot renew --dry-run, expect no error
23.16.3. manual plugin
- authenticator only
- obtain a certificate by giving you instructions to perform domain validation yourself. Additionally allows you to specify scripts to automate the validation task in a customized way. hooks can be specified
- Challenge types
- http-01, dns-01 or tls-sni-01
Example usage for HTTP-01:
certbot certonly --manual --preferred-challenges=http --manual-auth-hook /path/to/http/authenticator.sh --manual-cleanup-hook /path/to/http/cleanup.sh -d secure.example.com
authenticator.sh
#!/bin/bash dir=/path/to/web-root/.well-known/acme-challenge mkdir -p $dir filename=$dir/$CERTBOT_TOKEN test -f $filename || touch $filename echo $CERTBOT_VALIDATION > $filename chown -R www-data:www-data $filename
cleanup.sh
rm -f /var/www/htdocs/.well-known/acme-challenge/$CERTBOT_TOKEN
Environment vars available to hooks
- CERTBOT_DOMAIN
- The domain being authenticated
- CERTBOT_VALIDATION
- The validation string (HTTP-01 and DNS-01 only)
- CERTBOT_TOKEN
- Resource name part of the HTTP-01 challenge (HTTP-01 only)
- CERTBOT_CERT_PATH
- The challenge SSL certificate (TLS-SNI-01 only)
- CERTBOT_KEY_PATH
- The private key associated with the aforementioned SSL certificate (TLS-SNI-01 only)
- CERTBOT_SNI_DOMAIN
- The SNI name for which the ACME server expects to be presented the self-signed certificate located at $CERTBOT_CERT_PATH (TLS-SNI-01 only)
- CERTBOT_AUTH_OUTPUT
- (cleanup hook only) Whatever the auth script wrote to stdout
23.16.3.1. Example usage for DNS-01
(Cloudflare API v4) (for example purposes only, do not use as-is)
certbot certonly --manual --preferred-challenges=dns --manual-auth-hook /path/to/dns/authenticator.sh --manual-cleanup-hook /path/to/dns/cleanup.sh -d secure.example.com
authenticator.sh
#!/bin/bash # Get your API key from https://www.cloudflare.com/a/account/my-account API_KEY="your-api-key" EMAIL="your.email@example.com" # Strip only the top domain to get the zone id DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)') # Get the Cloudflare zone id ZONE_EXTRA_PARAMS="status=active&page=1&per_page=20&order=status&direction=desc&match=all" ZONE_ID=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=$DOMAIN&$ZONE_EXTRA_PARAMS" \ -H "X-Auth-Email: $EMAIL" \ -H "X-Auth-Key: $API_KEY" \ -H "Content-Type: application/json" | python -c "import sys,json;print(json.load(sys.stdin)['result'][0]['id'])") # Create TXT record CREATE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN" RECORD_ID=$(curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \ -H "X-Auth-Email: $EMAIL" \ -H "X-Auth-Key: $API_KEY" \ -H "Content-Type: application/json" \ --data '{"type":"TXT","name":"'"$CREATE_DOMAIN"'","content":"'"$CERTBOT_VALIDATION"'","ttl":120}' \ | python -c "import sys,json;print(json.load(sys.stdin)['result']['id'])") # Save info for cleanup if [ ! -d /tmp/CERTBOT_$CERTBOT_DOMAIN ];then mkdir -m 0700 /tmp/CERTBOT_$CERTBOT_DOMAIN fi echo $ZONE_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID echo $RECORD_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID # Sleep to make sure the change has time to propagate over to DNS sleep 25
cleanup.sh
#!/bin/bash # Get your API key from https://www.cloudflare.com/a/account/my-account API_KEY="your-api-key" EMAIL="your.email@example.com" if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID ]; then ZONE_ID=$(cat /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID) rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID fi if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID ]; then RECORD_ID=$(cat /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID) rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID fi # Remove the challenge TXT record from the zone if [ -n "${ZONE_ID}" ]; then if [ -n "${RECORD_ID}" ]; then curl -s -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \ -H "X-Auth-Email: $EMAIL" \ -H "X-Auth-Key: $API_KEY" \ -H "Content-Type: application/json" fi fi
23.16.4. certbot standalone plugin
An authenticator plugin uses a standalone webserver to obtain a certificate. It's useful when there's no webserver. Need to open port 80 or 443 http-01 or tls-sni-01 challenge.
sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install certbot
Certbot may run as a standalone webserver, allow either 80 or 443 sudo ufw allow 80
generete certificate only sudo certbot certonly –standalone –preferred-challenges http -d example.com
The certbot package we installed takes care of this for us by adding a renew script to /etc/cron.d
/etc/letsencrypt/renewal/example.com.conf renew_hook = systemctl reload rabbitmq
sudo certbot renew –dry-run
23.16.5. webroot plugin --webroot -w letsencrypt:certbot:webroot
- authenticator only
- obtain certificate and write to an existing webroot without webserver config mod
- Challenge types
- http-01
-w- alias
--webroot-path MYWEBROOT_PATH
certbot certonly --webroot -w /var/www/example/ -d www.example.com -d example.com -w /var/www/other -d other.example.net -d another.other.example.net
23.16.6. manage and renew certificates
/etc/letsencrypt/archive and /etc/letsencrypt/keys contain all previous keys and certificates, while /etc/letsencrypt/live symlinks to the latest versions.
# check status for all certificates certbot certificates certbot certonly --cert-name example.com # Update an existing certificate with a new certificate that contains all old domains and new domains # Use -d to specify all existing and new domains. certbot --expand -d existing.com,example.com,newdomain.com # remove some domains from an exisiting certificate. before it contains example.com and www.example.com certbot certonly --cert-name example.com -d example.com # overwrites all domains for an existing certificate certbot certonly --cert-name example.com -d example.org,www.example.org # revoke certbot revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem # renew any existing certificates that expire in less than 30 days. # --pre-hook and --post-hook hooks run before and after every renewal attempt. certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start" # If you want your hook to run only after a successful renewal, use --deploy-hook in a command like this. certbot renew --deploy-hook /path/to/deploy-hook-script
23.16.6.1. Hooks
certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start" # If you want your hook to run only after a successful renewal, use --deploy-hook in a command like this. certbot renew --deploy-hook /path/to/deploy-hook-script
deploy-hook-script
#!/bin/sh
set -e
for domain in $RENEWED_DOMAINS; do
case $domain in
example.com)
daemon_cert_root=/etc/some-daemon/certs
# Make sure the certificate and private key files are
# never world readable, even just for an instant while
# we're copying them into daemon_cert_root.
umask 077
cp "$RENEWED_LINEAGE/fullchain.pem" "$daemon_cert_root/$domain.cert"
cp "$RENEWED_LINEAGE/privkey.pem" "$daemon_cert_root/$domain.key"
# Apply the proper file ownership and permissions for
# the daemon to read its certificate and key.
chown some-daemon "$daemon_cert_root/$domain.cert" \
"$daemon_cert_root/$domain.key"
chmod 400 "$daemon_cert_root/$domain.cert" \
"$daemon_cert_root/$domain.key"
service some-daemon restart >/dev/null
;;
esac
done
Script files can be placed at /etc/letsencrypt/renewal-hooks/pre, /etc/letsencrypt/renewal-hooks/deploy, and /etc/letsencrypt/renewal-hooks/post Script files are run in alpha order.
23.16.6.2. Renewal conf
Sample /etc/letsencrypt/renewal/mydomain.com.conf
# renew_before_expiry = 30 days version = 0.21.1 archive_dir = /etc/letsencrypt/archive/mydomain.com cert = /etc/letsencrypt/live/mydomain.com/cert.pem privkey = /etc/letsencrypt/live/mydomain.com/privkey.pem chain = /etc/letsencrypt/live/mydomain.com/chain.pem fullchain = /etc/letsencrypt/live/mydomain.com/fullchain.pem # Options used in the renewal process [renewalparams] account = abcfdfsda installer = nginx authenticator = nginx
23.16.7. A certificate nginx:ssl:example
- Or see an example of Let's Encrypt on Nginx nginx:ssl:server block
server { # your original setting server_name .yoursite.com listen [::]:443 ssl ipv6only=on; # You might need to remove ipv6only=on; if it was set. managed by Certbot listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/yoursite.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/yoursite.com/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot # set proxy }
- certbot renewal config file
- /etc/letsencrypt/renewal/mydomain.com.conf
- global config
- /etc/letsencrypt/cli.ini
23.16.8. Windows IIS
- IIS 8 on Windows Server 2012 or IIS 8.5 on Windows Server 2012 R2.
- IIS 10 on Windows server 2016
Manual Install SSL on IIS 8 or IIS 10 :: create CSR on IIS, submit and get SSL, import cert.
- Or on IIS, import the .p12 file with key and crt from Linux
IIS Binding
- Server Binding
IP:Port:HostHeaderto a website. e.g.*:80:**:81:*192.168.1.1:80:**:80:www.microsoft.com*:80:microsoft.com- (no term)
- HTTPS is a server binding with SSL certificate
*:443:abc.com*:443:www.abc.com. For IIS 2012+, enableServer Name Indicationoption to bind multiple SSL certificates to a single IP address. Before, IIS can only bind a single SSL cert. to one IP address
IIS Websites
- Path (a container of physical and virtual directories) + Server Binding = Website. e.g. default container is
%systemdrive%\inetpub\wwwroot
win-acme (ACME client) Intro GitHub
Download latest release, unzip and run letsencrypt.exe with admin privileges.
- A task is added to Windows Task Scheduler
23.16.9. Alternative ACME clients
- https://www.digitalocean.com/community/tutorials/an-introduction-to-let-s-encrypt
- Automatic Certificate Management Environment. It is a protocol. Let's Encrypt just issues certificates
- Cerbot is one ACME client that is developed by Electronic Frontier Foundation (EFF) which uses ACEM protocol
- lego">Written in Go, lego is a one-file binary install, and supports many DNS providers when using the DNS challenge
- acme.sh">acme.sh is a simple shell script that can run in unprivileged mode, and also interact with 30+ DNS providers
- Caddy">Caddy is a full web server written in Go with built-in support for Let's Encrypt.
23.16.10. Limitation
- 5 levels of validation
- Domain Validation DV
- domain registrar or put a file on the domain
- (no term)
- Organization Validation OV
- Extended Validation EV
- phone call. Most expensive
- (no term)
- Wildcard Certificate
- Certificate Authority (CA) provides the above validation and Commercial Certificate Authorities are the only one provide EV and Wildcard
- Let's Encrypt provides DV only and planning to support Wildcard in near future. A certificate with up to 100 hostnames
23.16.11. SSL config nginx:ssl:config
23.16.11.1. Combine end-user and bundled intermediate certificates
cat your_domain.crt intermediate.crt root.crt >> ssl-bundle.crt # e.g. PositiveSSL cat example_com.crt COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt >> ssl-bundle.crt # e.g. combine end-user and bundled intermediate certificates cat example_com.crt bundle.crt >> ssl-bundle.crt
23.16.11.2. Strengthen Nginx SSL Setup
# Protocol support ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Perfect when only 1.2 is used ssl_protocols TLSv1.2; # Put it in server block ssl_dhparam ssl/dhparam.pem; # Generate DHE openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096 # Specify ciphers that are enabled on Nginx # Default ssl_ciphers HIGH:!aNULL:!MD5; # Change it to use openssl ciphers, run this to get ciphers provided by openssl and wrap it with double quotes and assign it to ssl_ciphers # openssl ciphers ssl_ciphers "AES256+EECDH:AES256+EDH:!aNULL;" ssl_prefer_server_ciphers on; # Default: ssl_session_cache none; # shared :: a cache shared between all Nginx worker processes. In bytes. 1 mb can store about 4000 sessions. le_nginx_SSL is a cache name and can be used in multiple virtual servers # Default: ssl_session_timeout 5m; ssl_session_cache shared:le_nginx_SSL:1m; ssl_session_timeout 1440m; # SSL Stapling (not implemented by Let's Encrypt) ssl_stapling on; ssl_stapling_verify on;
23.17. nginxconfig.io
- https://nginxconfig.io
- https://nginxconfig.io/?domain=myweb.ca&ssl_profile=intermediate&hsts=false&email=ssl@myweb.ca&non_www=false&wordpress
server { listen 80; listen [::]:80; server_name .myweb.ca; include nginxconfig.io/letsencrypt.conf; # letsencrypt.conf # location ^~ /.well-known/acme-challenge/ { # root /var/www/_letsencrypt; # } location / { return 301 https://www.myweb.ca$request_uri; } }
nano /etc/nginx/nginx.conf nano /etc/nginx/sites-available/myweb.ca.conf nano /etc/nginx/nginxconfig.io/php_fastcgi.conf nano /etc/nginx/nginxconfig.io/wordpress.conf nano /etc/nginx/nginxconfig.io/general.conf ln -s /etc/nginx/sites-available/myweb.ca.conf /etc/nginx/sites-enabled/myweb.ca.conf openssl dhparam -dsaparam -out /etc/nginx/dhparam.pem 2048 # create dir sudo -u www-data sh -c "mkdir -p /var/www/_letsencrypt"~ chown www-data:www-data /var/www/_letsencrypt # comment out all ssl_* directives - add `#;` in front sed -i -r 's/(listen .*443)/\1;#/g; s/(ssl_(certificate|certificate_key|trusted_certificate) )/#;#\1/g' /etc/nginx/sites-available/myweb.ca.conf # create cert only certbot certonly --webroot -d myweb.ca -d www.myweb.ca --email ssl@myweb.ca -w /var/www/_letsencrypt -n --agree-tos --force-renewal --dry-run # add back ssl_* directives - remove `#;` in front sed -i -r 's/#?;#//g' /etc/nginx/sites-available/myweb.ca.conf
23.18. deny nginx:d:deny
Syntax: deny address | CIDR | unix: | all;
Default: —
Context: http, server, location, limit_except
location ~* \.(git|rb|inc|ht)$ { deny all; }
23.19. Redirect a website to another website
23.19.1. HTTPS to HTTPS redirect
- Redirect https://mywebsite.com to https://mywebsite2.ca. Both websites are not hosted on this Nginx server
/etc/nginx/sites-available/mywebsite.com.confserver { listen 443 ssl http2; listen [::]:443 ssl http2; server_name mywebsite.com www.mywebsite.com; set $base /var/www/mywebsite.com; root $base/public; # SSL ssl_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/mywebsite.com/privkey.pem; ssl_trusted_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem; # index.php fallback # remove comment after certificate is installed #location / { # return 302 https://www.mywebsite2.ca; #} } # non-www, subdomains redirect server { listen 443 ssl http2; listen [::]:443 ssl http2; server_name .mywebsite.com; # SSL ssl_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/mywebsite.com/privkey.pem; ssl_trusted_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem; return 302 https://www.mywebsite2.ca; } # HTTP redirect server { listen 80; listen [::]:80; server_name .mywebsite.com; include nginxconfig.io/letsencrypt.conf; location / { return 302 https://www.mywebsite2.ca; } }
- Create an index.html
/var/www/mywebsite.com/public/index.html - Change DNS
A @andCNAME www or A wwwrecord formywebsite.comto point to Nginx- Go to
http://mywebsite.comandhttp://www.mywebsite.com, Nginx Welcome page should show - DNS is propagated!
- Go to
Run
ln -s /etc/nginx/sites-available/mywebsite.com.conf /etc/nginx/sites-enabled/mywebsite.com.conf sed -i -r 's/(listen .*443)/\1;#/g; s/(ssl_(certificate|certificate_key|trusted_certificate) )/#;#\1/g' /etc/nginx/sites-available/mywebsite.com.conf systemctl reload nginx # dry run to review errors certbot certonly --webroot -d mywebsite.com -d www.mywebsite.com --email ssl@gmail.com -w /var/www/_letsencrypt -n --agree-tos --force-renewal --dry-run sed -i -r 's/#?;#//g' /etc/nginx/sites-available/mywebsite.com.conf systemctl reload nginx
23.19.2. HTTP to HTTPS redirect nginx:redirect:HttpToHttps
- Or see an example of Let's Encrypt on Nginx nginx:ssl:server block
server { # we want to listen on port 80 on all IPs on our system - both IPv4 and IPv6 listen 80; listen [::]:80; # our primary server name is the first, aliases simply come after it. you can also include wildcards like *.example.com server_name source.com www.source.com source.ca www.source.ca; location = /special-page { return 302 http://www.target.ca; } location ~ ^/(news|magazine-archives) { rewrite (?i)\/([^\/]+)-(\d+)\/?$ https://www.target.ca/source/$1 last; } location / { return 302 https://www.target.com/citb; } # define our access and error logs for this vhost # access_log /var/log/nginx/access-sourceweb.log; # error_log /var/log/nginx/error-sourceweb.log; }
24. Git
24.1. Show config
# 3 levels: system < global (user, ~/.gitconfig) < local (./.git/config) < ./.git/info/attributes < ./.gitattributes # List all active config for the current folder git config --list | grep credential git config --get core.autocrlf # List global config only git config --global --list # Edit or check all global config git config --global --edit # Get all files that forms each config, also good to locate config files git config --list --show-origin # Edit system config file requires admin permission. Use notepad to change # C:\\ProgramData/Git/config # core.symlinks = false # edit system config # Windows :: C:/Program Files/Git/mingw64/etc/gitconfig # Mac Homebrew :: # - Source :: /opt/homebrew/etc/gitconfig # - Old :: /usr/local/etc/gitconfig git config --edit --system # edit global config which is ~/.gitconfig # Mac /Users/lili/.gitignore git config --edit --global # Find path to git.exe which git.exe
24.1.1. User name and email
# change username and email in global git config --global user.name "Li Li" git config --global user.email "a@b.ca" # change username and email for the current repo git config user.email myname@abc.omc git config user.name "My Name" # set temporary config for commit git -c user.name='Li Li' -c user.email='a@a.ca' commit -m '...' # or git commit -m '...' --author="Li Li <a@b.ca>"
Don't verify SSL and use temporary username and password to pull and push
git -c http.sslVerify=false -c user.name=abc -c user.password=xyz pull -p
24.1.2. .gitattributes
./.git/info/attributesor./.gitatrributes
# All files are checked into the repo with LF # text :: It marks the path as a text file, which enables end-of-line conversion: When a matching file is added to the index, the file’s line endings are normalized to LF in the index. Conversely, when the file is copied from the index to the working directory, its line endings may be converted from LF to CRLF depending on the eol attribute, the Git config, and the platform # -text :: Unsetting the text attribute on a path tells Git not to attempt any end-of-line conversion upon checkin or checkout. Equivalent to core.autocrlf = false # text=auto :: Git decides by itself whether the file is text or binary. If it is text and the file was not already in Git with CRLF endings, line endings are converted on checkin and checkout as described above. Otherwise, no conversion is done on checkin or checkout. * -text # These files are checked out using CRLF locally *.bat eol=crlf
# List files in the index and the working tree # --eol will show i/<eolinfo><SPACES>w/<eolinfo><SPACES>attr/<eolattr><SPACE*><TAB><file> # - <eolinfo> is the file content identification used by Git when the "text" attribute is "auto" (or not set and core.autocrlf is not false). <eolinfo> is either "-text", "none", "lf", "crlf", "mixed" or "". "" means the file is not a regular file, it is not in the index or not accessible in the working tree. # - <eolattr> is the attribute that is used when checking out or committing, it is either "", "-text", "text", "text=auto", "text eol=lf", "text eol=crlf". Since Git 2.10 "text=auto eol=lf" and "text=auto eol=crlf" are supported # - Both the <eolinfo> in the index ("i/<eolinfo>") and in the working tree ("w/<eolinfo>") are shown for regular files, followed by the ("attr/<eolattr>"). git ls-files --eol
24.2. Alias
# git co yourbranch git config --global alias.co checkout git config --global alias.br branch git config --global alias.st status git config --global alias.sti 'status --ignored' git config --global alias.acp '!llg_acp(){ git add . && git commit -m "$1" && git push; };llg_acp'
~/.gitconfig
[alias] co = checkout acp = "!llg_acp(){ git add . && git commit -m \"$1\" && git push; };llg_acp" commit-x = -c user.name='Change to X Author' -c user.email='x@a.ca' commit # Or include a separate file for alias [include] path = /Users/lili/.gitconfig.alias.conf # .gitconfig.alias.conf [alias] # ...
24.3. Working Tree
- Working Copy (unstaged, working directory, working tree) ->
git add-> Index (staged) ->git commit->HEAD - ">
cat .git/HEAD- It can be the tip of a branch
- Or any commit in any branch
- Use
git checkout <branchname>to set HEAD to the tip of a branch
- Use
- the previous commit
- it was never staged nor committed
- it was previously committed, but it is not staged
- ready for commit
- file is changed but not staged
24.4. Archive
Zip to a file
git archive [--format=<fmt>] [--list] [--prefix=<prefix>/] [<extra>] [-o <file> | --output=<file>] [--worktree-attributes] [--remote=<repo> [--exec=<git-upload-archive>]] <tree-ish> [<path>…]
git archive --format=zip -o /full/path/to/zipfile.zip master
- d:tar or zip
- prepend
<prefix>/to each filename in the archive -o <file>,--output=<file>Export files from master branch to a folder without compression
git archive master | tar -x -C /export/to/thisfolderwithoutcompression
24.5. diff
- git:stages
- Between Working Copy and Index (Staged)
- Between HEAD and Index (Staged)
- Between (Unstaged or Staged) and HEAD
- Between (Unstaged or Staged) and HEAD^
- git:branch:file difference
24.6. Patch
- Create a patch
- Refer to git:diff
git diff > my.patchgit diff --cached > my.patchgit format-patch -10git fortmat-patch -10 --stdout > my.patch
- Apply a patch
- Apply a patch to working directory
git apply my.patch
24.7. Show file names only between 2 commits
Show file names only for 1 commit
git diff-tree --no-commit-id --name-only -r "<SHA>"
Show file names only between 2 commits
git --no-pager diff --name-only SHA1 SHA2
24.8. Stash, Reset, Clean
- git:stages
git reset- eq. to
git reset --mixed HEAD- Reset current HEAD to a
<tree-ish>or<commit>which defaults toHEAD - default mode. Reset the Index (staged) but not the working tree
- Reset current HEAD to a
- eq. to
- See 1.6
- ">remove untracked files
- Save and remove Staged and Unstaged changes
- Stash tracked file changes
git stash- Give a name
git stash save 'my stash'- Stash tracked file changes and untrack files
git stash --include-untracked
- List all stashes
git --no-pager stash list- Apply latest stash
git stash apply- Apply 2nd latest stash
git stash apply "stash@{2}"- (no term)
- Apply when there's no staged and unstaged
- (no term)
- After applied, all changes are unstaged
- Remove latest stash
git stash drop "stash@{0}"- remove all stashes
git stash clear
24.9. Checkout vs Reset vs Revert
- Current branch
releasehas c1, c2 and c3 commits in ordergit checkout c1- sets branch to commit 1 in a detached HEAD state. HEAD is not at tip of branch. It's at that commit. Not changing anything
- You can go back to the branch HEAD by
git checkout release. Then it's no longer in detached HEAD state.
- You can go back to the branch HEAD by
git checkout <branchname>- go back to the tip of the branch
git checkout c1 .orgit checkout -f c1 -- .- The working directory is set to the same as c1. Changes to be committed are the changes needed to go back from c3 to c1. The HEAD is still at
c3. If you commit as c4 (which is after c3), c4 will be the same as c1.- To undo this,
git reset --hard
- To undo this,
git checkout -- path/to/file- discard changes for a file and revert the file to the last committed
- Checkout a file from another branch
git checkout mybranchname -- path/to/file
git reset c1- sets branch HEAD to commit 1
- Commits beyond c1 to the previous HEAD are saved in reflog
- (no term)
git reset --hard- eq. to
git reset --hard HEAD - clean working tree, go back to HEAD but the previously untracked changes are intact
- eq. to
git reset --hard HEAD^- set branch HEAD to the second last commit
git reset HEAD^- set branch HEAD to the second last commit, the last commit changes are in working tree
- (no term)
git resetrewrites history. Only use it for local changes that are never pushed to remotegit revert c2 --no-edit- undo commit c1 and creates a commit.
--no-editmeans to skip the commit message- https://mirrors.edge.kernel.org/pub/software/scm/git/docs/howto/revert-a-faulty-merge.txt
- If c2 is a merge commit, it will have more than 1 parents
Running
git log c2shows all sub commits that are merged inc2merge commit.git log c2 -1only shows the merge commit. Something like this showscommit c2 Merge: parent1 parent2 Author: Li Li <li@li.com> Date: Mon Jun 24 11:36:52 2024 -0400 Merge branch 'feature-branch' into release
In this case, parent1 is original commit in the feature-branch, parent2 is c1 in the current release branch, but it is better for you always check. To revert c2 on release branch,
git revert c2 -m 1where-m 1is to choose parent1
24.10. Checkout a folder from another branch to current branch
current branch is master and get folder target-folder/ from feature branch
git checkout master git checkout feature target-folder/
24.11. Commit
- Amend last commit message
git commit --amend -m "new commit message"- Amend last commit's author
git commit --amend --author="FirstName LastName <a@b.ca>"- After changing author, it still keeps who initially makes the commit
- (no term)
If the commit is not the last commit on HEAD.. Do this but it will change history from that commit onwards
# bbc643cd is the commit that you want to change, ^ means rebase back to the commit before the one you wish to modify git rebase --interactive 'bbc643cd^' # for commit bbc643cd, change `pick` to `edit` # you can pick more commits to edit/amend # then modify author git commit --amend --author="MA Authors <li@mortgageautomator.com>" --no-edit git rebase --continue # rebase will stop at the next commit that you previously mark as edit # rebase continue until it ends git push -f
24.12. Branch
24.12.1. Current branch info
# all local branches and their last commit message, current branch is * highlighted git branch -v # all local branches and their last commit message and the commit's remote branch (upstream) git branch -vv # current branch and its upstreams # -s :: short # -b :: branch and its tracking info (upstream) # This shows the current local branch has a corresponding upstream branch # ## release..origin/release # This shows the current local branch does not have a corresponding upstream branch # ## release git status -sb
24.12.2. Duplicate a remote branch as a new local branch
git checkout -b my-new-local-branch origin/my-remote-branch
24.12.3. Duplicate current branch with uncommited changes and switch to it
git checkout -b newbranch
Create a local branch and then push to a new remote branch. Save changes to a new branch.
git checkout -b newbranch git add . && git commit -m "..." git push --set-upstream origin newbranch # eq. git push -u origin newbranch # eq. set current local branch's upstream without specifying the local branch name git push -u origin HEAD
24.12.4. Duplicate current branch to a commit and switch
git checkout -b newbranch <SHA1>
24.12.5. Delete branch
git checkout master- you can't be on the branch that is going to be deleted
- (no term)
git branch -d branchgroup/branchname- delete the local branch only if it has been pushed and merged with the remote branch
- (no term)
git branch -D branchname- delete the local branch even if it hasn't been pushed or merged with the remote branch
- (no term)
- Undo branch delete
git branch branchName <sha1>- the sha1 shows after the branch is deleted
- (no term)
git push origin --delete branchgroup/branchname- delete a branch remotely
- ~git push origin :branchgroup/branchname
error: unable to push to unqualified destination: remoteBranchName The destination refspec neither matches an existing ref on the remote nor begins with refs/, and we are unable to guess a prefix based on the source ref. error: failed to push some refs to 'git@repository_name'- This error shows if the remote branch is already deleted
git fetch -p--prune, -ppurge/prune. Before fetching, remove any remote-tracking references that no longer exist on the remote- On local there's
origin/abcbut remote does not have branch abc, then localorigin/abcwill be removed
- On local there's
24.12.6. Rename branch
- Branch name can contain
-- Restrain from using
,.
- Branch name better not have
:
# Rename current branch git branch -m new-branch-name # Rename another branch which is not the current branch git branch -m old-branch-name new-branch-name # If the previous local branch has pushed to remote, do the above to rename the local branch, and do these # Delete the old remote branch and push the current branch to the new remote branch git push origin :old-branch-name new-branch-name # Reset the upstream of the local branch to the new remote branch git push origin -u new-branch-name
24.12.7. Find branches which contain a commit
git branch --contains mycommithash
24.12.8. Commit difference between 2 branches
- What commits
lihas butmasterdoesn't git log master..li- For a specific file
git log master..li path/to/file- Current branch
git log master..HEADgit log master..@
For multiple branches, e.g. to see what commits in (feature-1-branch and feature-2-branch) but are not on (release nor master)
git log --no-merges feature-1-branch feature-2-branch ^release ^master- To limit to a file
git log --no-merges feature-1-branch feature-2-branch ^release ^master -- path/to/file
- For a specific file
- (no term)
- What commits a)
lihas butmasterdoesn't and b)masterhas butlidoesn'tgit log master...li
- (no term)
- Options
-p- show changed files for each commit listed
git log -p master..li
- (no term)
- See git:branch:file difference
- (no term)
- See 24.14
24.12.9. File difference between 2 branches
- git:diff
git diff --name-status master..origin/master- D or -
- Missing file or line in remote branch
- A or +
- New file or line in remote branch
- M
- Both are modified
git diff --stat --color master..origin/master- Only number of lines of difference, can't see which files are D (deleted), A (added) or M (modified)
- See git:log:commit difference
24.12.10. Find authors of all remote/local branches and tags
git for-each-ref --format='%(committerdate) %09 %(authorname) %09 %(refname)' | sort -k5n -k2M -k3n -k4n
24.12.11. Make current branch exactly as another branch
Make current branch li exactly the same as remote branch origin/master.
git checkout li git reset --hard origin/master
Pull forced after push forced
git checkout li git rebase main git push -f # On another computer git checkout li git fetch git reset --hard origin/li
24.12.12. Force push to remote branch
local master has different commits than remote li's master branch
git checkout master git push li master -f
24.12.13. Find most recent ancestor of 2 branches. See 24.14
24.12.14. Revert to common ancestor
You have a feature branch and master branch. Both have remote branches and are in sync. Branch feature has some commits that master doesn't have and master are ahead of feature.
Keep the commits in feature but undo those commits and make a new commit in feature.
Find the most recent ancestor of feature and master git merge-base feature master
Say the commit is common1 Make a new branch to test if it's a real common ancestor git checkout feature git checkout -b featuretest common1 git log master..featuretest // You should see nothing and featuretest is not ahead of master
Then make feature to go back to that commit without changing history
= Method 1 =
git checkout feature
git checkout common1 . or git checkout -f c1 --.
git commit -m "reset to common1"
Now you can merge master into feature
git merge –no-commit master
= Method 2 =
/ Revert commits one by one
/ Get commits from an old commit to HEAD
git log common1..HEAD
// or
git rev-list common1..HEAD
git revert –no-commit D
git revert –no-commit C
git revert –no-commit B
…
git commit -m "reset to common1"
= Method 3 =
// Caution! You are rewriting history on feature branch at local and remote
git checkout feature
git reset –hard common1
git push -f
24.12.15. Find branches merged into a branch
# List local branches merged into master # Use `-a` to show both local and remote branches # Use `-r` to show only remote branches git branch --merged master # List local branches merged into HEAD (i.e. tip of current branch) git branch --merged # List branches that have not been merged git branch --no-merged
24.12.16. Pull from remote/master to master branch which is not the current branch
git fetch origin master:master- Currently at local branch
devand need to pull remote/master to local master branch - This does not work if current branch is local master
- Don't use
git pull origin masteras it pulls fromorigin/masterto current localdevbranch - update all
origin/branchnamebranches - update all remote branches for all remotes
- Currently at local branch
24.12.17. Orphan Branch with no parent
24.12.18. git:upstream
24.13. Merge
- By default without any options, merge will not create merge commits when fast-forward is possible
--no-ff- Default (
--ff) - create a merge commit only if the merge cannot fast forward
--ff-only- Merge when fast-forward is possible, otherwise refuse to merge
--no-ff- create a merge commit in all cases
--no-commit- stop just before creating a merge commit. There is no stop if fast-forward is possible.
- Fix the conflict and
git addfiles then normal commit
- Fix the conflict and
- Default (
- Resolve conflict
- Conflict markers
<<<<<<<=======>>>>>>>
git add .git commit -m ""orgit merge --continue
- Conflict markers
git merge --abort- Combine all integrated changes into a single commit instead of preserving them as individual commits. Need to make a normal commit after
--squash
24.13.1. Merge just one file
Merge file f of branch B's HEAD into f of branch A~
git checkout A git checkout B f # --patch option makes it interactive # if f on B does not exist yet, omit --patch, otherwise, you'll get a "No Change" message # B could be any commit
24.14. merge-base
# Return the most recent ancestor of 2 branches git merge-base branch2 branch3 # Return the best common ancestor(s) between 2 commits # If the return is identical to last-commit-feature-branch, then last-commit-feature-branch is in previous commit before last-commit-master-branch git merge-base <last-commit-master-branch> <last-commit-feature-branch> # Let's say master branch is tagged every time a deployment to live happens, let's say a tag is called `latest`, find that commit on master git show latest # copy this hash, let's assume it's last-commit-master-branch # you have a commit abc, you want to find out if it is already deployed git merge-base last-commit-master-branch abc # eq. to git merge-base master abc git merge-base $(git log -1 --pretty=format:"%h" master) abc # If it returns abc, then abc is deployed # To get last commit of master branch git log -1 master
24.15. Rebase
24.15.1. Rebase backup and revert
# rebase saves your starting point to ORIG_HEAD so this is usually as simple as: git reset --hard ORIG_HEAD # However, the reset, rebase and merge all save your original HEAD pointer into ORIG_HEAD so, if you've done any of those commands since the rebase you're trying to undo then you'll have to use the reflog. # Referrence: https://stackoverflow.com/questions/134882/undoing-a-git-rebase
Old method
# Before doing rebase, tag the head of the current branch and easily reset to that after rebase if there's anything wrong # If it shows fatal error, that's good! LILIBACKUP is not used git show LILIBACKUP # Create a tag to point to the current head commit git tag LILIBACKUP # do git rebase # If there's anything wrong after git rebase is done, reset back to it git reset --hard LILIBACKUP # Delete the local LILIBACKUP tag git tag -d LILIBACKUP
24.15.2. Rebase master onto feature branch
git checkout master git pull --rebase # or just git pull if you're sure local master can be fast forwarded git checkout feature # make sure local feature is up-to-date with remote feature git pull # make some changes and git commit # find files that are different between local feature and local master # refer to git:branch:file difference git diff --name-status master..feature # A status :: feature branch has a new file that master does not exist git rebase master # git status to review conflicts if exist. After resolving, git add . git rebase --continue # there may be several rounds of continue # if you see working tree is clean after `git status` and continue can't continue: git rebase --skip git status # review and resolve conflicts as above git rebase --continue # now your commits in feature are applied on top of the master tip # See file difference between master and the feature branch, these should list all changed files in feature branch git diff --name-status master..feature # The feature branch's new commits are on top of the master branch tip git log # check file difference between local feature and master to see if that's the change you want it # you may need to manually make a commit on local feature to fix the rebase # git add . && git commit -m "fix rebasing" # you can merge commits into one commit on feature branch to further clean up your feature branch commits git checkout master git merge feature
The result is feature and master are exactly the same: C2 -> C3 -> C4
24.15.3. Rebase master onto a feature branch that has merges from master
my-branchis based onmasterand it has its own commits, merges from master and conflict resolves from time to time- The result we want is: modify
my-branchhistory so that there's only one commit on top of themasterbranch- Benefits of using
git merge --squash- We don't have to resolve the conflicts we previously resolved
- You can review all the changes previously made
- Caveat of using
git merge --squash- Changes are moved to working tree and waiting for you to index and commit
- Benefits of using
# After many merges from master to my-branch, now we want to rebase master onto my-branch: git checkout my-branch git branch -m my-branch-old git checkout master git checkout -b my-branch git merge --squash my-branch-old git commit git branch --set-upstream-to origin/my-branch git push -f
You want to see what changes are made for branch-for-cr
git checkout branch-for-cr git pull --rebase=i # This is to make sure local branch-for-cr is truly the same as remote because remote history might be changed # - If the rebase prompts to select/pick commits, pick all commits (by doing nothing) `:wq` # - `git rebase --skip` for all of them # Now local branch-for-cr is exactly the same as remote branch-for-cr git checkout master && git pull git checkout -b t-branch-for-cr git merge --squash branch-for-cr git add . && git commit -m "for CR" # Review the changes in the new commit
24.15.4. Rebase a base feature branch onto another feature branch
feature-bdepends onfeature-aand it should contain all changes fromfeature-afeature-acommit history is changed constantly- It's always on top of
masterbecausemasteris continuously rebased ontofeature-a feature-amightgit merge --squashintomasterwhenfeature-ais merged intomaster. See 24.15.3
- It's always on top of
feature-bis always on topfeature-a
- See https://softwareengineering.stackexchange.com/a/351790/294454
# Start feature-b git checkout feature_a git checkout -b feature_b # feature_b has some updates git add . && git commit -m "feature_b update" # feature_a now has more commits # feature_a may rebase master git checkout feature_a git rebase master # feature_a is merged to master # feature_a is squash merged to master git checkout feature_a git branch -m feature_a_old git checkout master git checkout -b feature_a git merge --squash feature_a_old git add . && git commit -m "Merge feature_a into master" git branch --set-upstream-to origin/feature_a git push -f # Every now and then, we need to rebase feature_a onto feature_b # Pull changes for feature_a (history might change) git checkout feature_a git pull --rebase=i # This is to make sure local feature_a is truly the same as remote because remote history might be changed # - If the rebase prompts to select/pick commits, pick all commits (by doing nothing) `:wq` # - `git rebase --skip` for all of them # Now local feature_a is exactly the same as remote feature_a git checkout feature_b git rebase feature_a --interactive # `:wq` to pick every commit # Scenario #1: # feature_b before rebasing feature_a: (M1-M2-A1-A2)-B1-B2 # feature_a before: M1-M2-A1-A2 # feature_a after: M1-M2-M3-A1'-A2'-A3 git rebase feature_a --interactive # Find the first commit of feature_b, which is B1 # `:wq` to pick all commits to replay # - Any commits that have conflict (from A1 to A2 on feature_b) and possibly any B1-B2 # - Pick all of them # `git rebase --skip` for any commits that before B1. Like (A1-A2) # Don't skip any B1-B2 # feature_b after rebasing feature_a: M1-M2-M3-A1'-A2'-A3-B1'-B2' # Summary: mark any commits before B1 (M1-M2-A1-A2) to skip in case `git rebase` prompts # Scenario #2: # feature_b before rebasing feature_a: (M1-M2-A1-A2)-B1-B2 # feature_a before: M1-M2-A1-A2 # feature_a after: M1-M2-M3-A4 git rebase feature_a -i # Find the first commit of feature_b, which is B1 # `:wq` to pick all commits to replay # `git rebase --skip` for any commits that are before B1. Like (A1-A2) # Don't skip any B1 and onwards # feature_b after rebasing feature_a: M1-M2-M3-A4-B1'-B2' # Finally, as soon as feature-a is merged into master git checkout master git pull git checkout feature_b git rebase --onto master feature_a feature_b
24.15.5. Interactive: Commands p(pick), s(quash), e(edit)
:cqto quit Vim without saving. Which means abort the current interactive rebase
24.15.6. Branch preference
git checkout feature-a # favor feature-a changes over master git rebase -Xtheirs master # favor master changes over feature-a git rebase -Xours master # -Xxxx is reversed in git merge!
24.15.7. Squash commits into one commit
- Branch feature has 4 commits ahead of branch master
- m1 > c1 > c2 > c3 > c4
- (no term)
- Combine c2, c3 and c4 (c1 is not included!) to c5 and assign a different commit message
# interactive rebase for commits NEWer than c1 git rebase -i c1 # open up VIM, keep the first line unchanged pick c2 and change pick to `s` or `squash` for all other commits pick c2 s c3 s c4 # `ESC` and `:wq` to save, VIM returns that what will happen. Comment lines with # and give a new commit message without # # a new SHA commit c5 is created with a commit msg that is specified # `ESC` and `:wq` to save. Done!
Commands other than p(ick), s(quash):
- e(dit)
- make file changes,
git add myfileand to continue to next command or commitgit commit --amend
24.15.8. Reorder commits
- https://stackoverflow.com/questions/2740537/reordering-of-commits
- a, b, c, d, e, f, g
- squash a, d, e and g
- Reorder
- c, [a + d + e + g], [b + f]
git checkout branchA # New commits including a git rebase -i a^ # Inital order # p a # p b # p c # p d # p e # p f # p g # Reorder # p c ... # p a ... # s d ... # s e ... # s g ... # p b ... # s f
24.15.9. Split a commit
git rebase -i c123^- Assign
eoreditto the commitc123aftergit rebaseis run,Esc :wqto start the rebase process git reset HEAD^git statusshowsLast command doneedit c123 some commit message
git add . && git commit -m "message"
git rebase --continue
24.16. Cherry pick
24.16.1. Example
Cherry pick commit f from Feature to master
a - b - c - d Master
\
e - f - g Feature
git checkout master git cherry-pick f
After
a - b - c - d - f' Master
\
e - f - g Feature
Let's say master has a new commit h
a - b - c - d - f' - h Master
\
e - f - g Feature
After git checkout feature && git rebase master, it becomes
a - b - c - d - f' - h Master
\
c - d - f' - h - e - g Feature
The original f is dropped
24.17. Show child commits for merge commit
After merge or rebase happened, a merge commit is created. git status gives :: C0 > C1 > merge commit abc
See what this merge commit abc consists of. Take a look at the git status of abc, you will find Merge: C1 xyz
Show file names only :: git diff –name-only C1..abc Show commits in a merge commit :: git log C1..abc Show file names in each commit in a merge commit :: git show –name-only C1..abc Show changes of a file that is caused by the merge commit :: git diff C1..abc – path/to/file
24.18. Fork
Sync a fork with original repo Add a remote: upstream
git clone git@github.com:YOUR-USERNAME/YOUR-FORKED-REPO.git
cd into/cloned/fork-repo
git remote -v
git remote add upstream git://github.com/ORIGINAL-DEV-USERNAME/REPO-YOU-FORKED-FROM.git
git remote -v
git fetch upstream
git pull upstream master
24.19. Symlink
core.symlinksdefaults to true- Git detect if the current filesystem supports symlink (FAT does not support) during
git cloneandgit init- if not, it will set it to false on the project folder
- Git detect if the current filesystem supports symlink (FAT does not support) during
- When it is false, the symlinks will be created as plain text file. To pull from remote again and make those plain text files to be symlinks:
- Keep
core.symlinks=false, create 3 and remove tracking for those symlink filesgit update-index --assume-unchanged symlink1using git:update-indexTo find all symlink files but are not created as symlinks on Windows:
git ls-files -s | awk '/120000/{print $4}'- Change it to true for the current repo
git config core.symlinks true
- and then
git pull
- Keep
24.20. Remove tracking for commited files
- A file is commited but you don't want to track it anymore in the future
git update-index --assume-unchanged path/to/file.txt- Remember the file will still be in Git but future clone won't grab the file
- If you change your mind and want to track it again
git update-index --no-assume-unchanged path/to/file.txt
- Don't track a folder of files which are already commited
- First try to run this because this will work with file names with space
git ls-files -z myFolderToIgnore/ | xargs -0 git update-index --assume-unchanged- Path is relative
- If you encounter error
xargs: git: bad file number, run thisgit ls-files -- myFolderToIgnore/ | xargs -l git update-index --assume-unchanged
- First try to run this because this will work with file names with space
- List all assumed unchanged files
git ls-files -v | grep '^[[[[:lower:]]]]'
- If repo has new commits for locally assumed unchanged files, Git will prevent from merging/pulling
- set it to
--no-assume-unchanged, stash the change or remember the change, merge/pull, and change to assumed-unchanged, and apply the previous change
- set it to
24.21. Remove a commited file from file but leave in filesystem
Delete a file from Git (top of the current branch) but leave the file in filesystem The file becomes Untracked File
git rm --cached file.txt # remove a folder git rm -r --cached .emacs.d/
Add file to local or global ignore so that the file won't appear as Untracked File and prevent from staging the file when git add .
24.22. Remove large files and folders
- Reference
- https://docs.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository
git gc # Get the top 10 biggest files. It will take about a minute for it to run git rev-list --objects --all | grep "$(git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -10 | awk '{print$1}')" # path/to/bigfile1, path/to/bigfile2 # filter-branch runs on the current branch and filter through from top (HEAD) to oldest git filter-branch --index-filter 'git rm --cached --ignore-unmatch path/to/bigfile1' HEAD # repeat for every file # To remove a directory git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch path/to/folder1' HEAD # To remove files and subdirectories (except hidden files) of a directory git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch sites/default/files/*' HEAD # filter-branch creates backups of your original refs namespaced under ~refs/original~ # Run this to delete the backed up refs, allowing the large objects to be garbage collected git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d # if all big files are in one branch only, just delete the branch will remove all the references # git branch -D branch-name # Prune all reflog references from the branch on back. # git reflog expire --expire=now branch-name # Prune all reflog references from now to the first commit git reflog expire --expire=now --all # Repack the repo by running gc and pruning old objects git gc --prune=now # Push changes back to remote repo. All branches (refs under ~refs/heads~) # I found I can push one branch only git push --all --force # Make sure tags are current and pushed to remote git push --tags --force
- –index-filter
- modify a repo's staging
- –cached
- remove a file from the index not the disk
- –ignore-unmatch
- prevent
git rmfrom failing if the file isn't there - HEAD
- to remove from the starts
Say git filter-branch is run once, and the backup refs/original is not deleted, the subsequent filter-branch will not run
Add -f to force it to run and overwrite refs/original
git filter-branch -f --index-filter [...]
git filter-branch [...] HEAD- just shrinks the current branch and leave the other branches intact.
- Say feature branch is a copy of master, then run filter-branch on feature. master branch and others are not affected. But now feature branch diverged from master.
- The get-top-10-biggest-files command returns the same files after filter-branch and the cleanup mentioned above are done. Because it's just one branch that you cleaned up, big files are still in other branches.
- You won't notice a big change in total git:repo size, as it's just one branch that had the big files removed.
- I didn't try
git push --forceto update a real remote branch.- Let's say the repo is at /path/to/repo, and you duplicate
masterbranch toshrinksizebranch. Run filter-branch and the cleanup. Then you create another folder /path/to/duplicaterepo, and clone only the
shrinksizebranch from the original repocd /path/to mkdir duplicaterepo cd duplicaterepo git clone file:///path/to/repo --branch shrinksize --single-branch cd repo du -sh .git/objects # push branch shrinksize to another remote as master branch git remote add anotherremote ... git push anotherremote shrinksize:master
- Let's say the repo is at /path/to/repo, and you duplicate
24.23. Remove one file from all remote branches
git filter-branch --force --index-filter \ "git rm --cached --ignore-unmatch ./path/to/file" \ --prune-empty --tag-name-filter cat -- --all # --prune-empty :: some filters will generate empty commits that leave the tree untouched. This instructs git-filter-branch to remove such commits if they have exactly one or zero non-pruned parents; merge commits will therefore remain intact # --tag-name-filter <command> :: This is the filter for rewriting tag names. The original tags are not deleted, but can be overwrriten; use "--tag-name-filter cat" to simply update the tags. # wildcards can be used e.g. ./path/to/*.xml # To remove a directory: git rm --cached --ignore-unmatch -r ./path/to/folder # Add the file to .gitignore echo "path/to/file" >> .gitignore git add .gitignore git commit -m "add path/to/file to .gitignore" git push origin --force --all --dry-run git push origin --force --all git push origin --force --tags --dry-run git push origin --force --tags
24.24. log - Show Commit History
- Syntax
git log [<options>] [<revision range>] [[--] <path>…]- (no term)
Commit formatting
git log --oneline git log --all --decorate --oneline --graph
- (no term)
- Options
--decorate[=short|full|auto|no]- default is short, just displays the commit hash
- (no term)
--pretty[=<format>],--format=<format>medium- default
oneline<hash> <title-line>- (no term)
- Placeholders
%as- author date YYYY-MM-DD
%s- subject
- (no term)
-- path/to/file/or/directorgit log origin/master -- path/to/a/file
- (no term)
- show commits on origin/master which modify a file
git log -- path/to/a/file
- (no term)
--branches[=<pattern>]- The pattern should be any local branches (not remote branches) which are any of
.git/refs/heads/* - If pattern is empty, commits of all local branches are included
- The pattern should be any local branches (not remote branches) which are any of
- (no term)
-all- All commits of all local and remote branches are included which are all of
.git/refs/*
- All commits of all local and remote branches are included which are all of
- (no term)
- See git:log:commit difference
git log -3- Last 3 commits
git log -3eq.git log -n 3git log --max-count=3
git log commit1..commit2 --format=oneline | wc -l- Count commits in a commit range, returns 1
git log --grep=<pattern>- Search commit message
%s- subject (commit message)
- (no term)
Show commits for multiple repos between start/end dates
#!/bin/bash TMPLOG=/mnt/e/project-logs.csv touch $TMPLOG echo "" > $TMPLOG gitrepos=( '/mnt/e/li/repo1' '/mnt/e/li/repo2' ) for i in "${gitrepos[@]}"; do if [ -d $i ] && [ -d $i/.git ]; then cd $i git log --since='last 2 months' --pretty=format:"$i,%ai,%s" >> $TMPLOG # After and including May 1, 2018 # git log --after='2018-5-1' echo "" >> $TMPLOG cd .. fi; done
- Show commits of a specific branch
git log origin/master- (no term)
- Show an author's commits
- Current branch
git log --author="Li Li" --before="2018-12-01" --after="2018-11-01"
- All branches include remote and local
git log --author="Li Li" --after=yesterday --all
--author- may use email
--author='Li*'
- match the regular expression limiting patterns without regard to letter case
--since=<date>,--after=<date>- Today
--after=midnight- yesterday
--after=yesterday- (no term)
- Commit date
- a commit's creation date
git log --author='Li Li' --format='%as %s' | grep -E ^2022-08-18
- Current branch
- Show an author's commits of a specifc branch
git log origin/master --author="Li Li"- Show an auhtor's commits by number of files changed, insertion and deletion
git log --author="Li Li" --oneline --shortstat- Show an author's commits by lines added/deleted for each file
git log --author="Li Li" --pretty=tformat: --numstat- (no term)
Exclude sub folders and files or include
# -- <path> :: limit commit history by path git log -- . ":(exclude)subfolder" # case-insensitive git log -- . ":(exclude,icase)wp-content/plugins" # exclude multiple git log -- . ":(exclude,icase)wp-content/plugins" ":(exclude,icase).idea" # include only git log -- ":(icase)wp-content/themes" ":(icase)wp-content/plugins" # include and exclude git log -- ":(icase)wp-content/themes" ":(exclude,icase)wp-content/themes/abc"
- (no term)
Show an author's commits by total lines added/deleted
git log --author="Li Li" --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }' -
24.24.1. git-when-merged
- https://github.com/mhagger/git-when-merged
brew install git-when-merged
# Find the merge commit that first brought the commit into the master branch # -l :: show the merge commit's message git-when-merged -l a-commit-hash master # -r :: follow the merge commit recursively git-when-merged -lr a-commit-hash master
24.25. Ignore, Local/Global Ignore, Show Ignored Files
- https://www.gitignore.io/
- Refer to git:update-index
git status -u- Add file or directory paths to
.git/info/exclude/(will not be committed).gitignorefile (will be commited)- Create
touch .gitignore
~\.gitignore_globaland rungit config --global core.excludesfile ~/.gitignore_globalgit config --list --show-origin- Sometimes, Windows is set up for all applications to use user profile folder at a different location, e.g. phpStorm
- Change excludesfile to a folder location that you know under
[core]aftergit config --global -e(edit the global config file) - Or try this
git config --global core.excludesfile K:/.gitignore_global
- Change excludesfile to a folder location that you know under
- Sometimes, Windows is set up for all applications to use user profile folder at a different location, e.g. phpStorm
git status --ignoredgit check-ignore -v path/to/filename- After 1.8.2,
**can be used to mean zero or more sub-directories (relative to the place where .gitignore is)bin/ignores any folder named bin/bin/ignores one folder named bin relative to the .gitignore fileHowever, if there're more than 1 folder,
wp-content/uploads/only ignore /wp-content/uploads. Use**/wp-content/uploads/to ignore any folder that has pathwp-content/uploads**/.DS_Store /main/**/bin/
Negate. An optional prefix
!which negates the pattern; any matching file excluded by a previous pattern will become included again. If a negated pattern matches, this will override lower precedence patterns sources.# Ignore everything * # But don't ignore these files... !.gitignore !script.pl !template.latex # etc... # ...even if they are in subdirectories. (don't ignore first level subdirectories) # without this line, script.pl will only be included in the root directory !*/
Ignore a folder but keep files in subdirectories, create
.gitignorein that folder# ignore everything * # add gitignore !.gitignore # keep sql files !*.sql # keep subdirectories !*/
e.g. ignore a directory site[123]
site\[123\]
*~- Ignore a commited file but leave it in filesystem
- Add the file to
.gitignore git rm --cached file.txtgit rm -r --cahced .emacs.d/- This way the file won't appear as Untracked File and prevent from staging the file when
git add .
- Add the file to
- https://www.atlassian.com/git/tutorials/gitignore
24.25.1. WordPress .gitignore git:gitignore:wordpress
*~ cgi-bin/ .well-known/ .DS_Store .svn .cvs *.bak *.swp Thumbs.db wp-config.php wp-content/uploads/ wp-content/blogs.dir/ wp-content/upgrade/ wp-content/backup-db/ wp-content/advanced-cache.php wp-content/wp-cache-config.php wp-content/w3tc-config/* wp-content/cache/* wp-content/cache/supercache/* *.log error_log wp-content/plugins/error_log wp-content/cache/ wp-content/backups/ sitemap.xml sitemap.xml.gz sql-admin/ cs-devops/
24.25.2. NodeJS, NPM .gitignore
node_modules npm-debug.log
24.26. Customize Settings per folder
For a single repository, attributes should be placed in $GIT_DIR/info/attributes file.
$GIT_DIR is the repository. But this file won't be commited.
If you want to commit, .gitattributes file in the root $GIT_DIR or any sub folders.
# Syntax # pattern attr1 attr2 ... *.txt text*.js text*.html text*.md text*.json text*.svg text # text is an attribute without value. `Set [attr] to true` *.pdf -text # text is again an attribute without value but prefixed with -. `Unset [attr] or Set [attr] to false` *.txt text eol=lf # Set text to true and Set eol to a value lf
24.27. Git Windows
gitk- GUI
start path/to/filename- to tell Windows to open a file
- (no term)
- edit a file using vi
vi afile - Run tree command
cmd //c tree- (no term)
- At Git Bash, open the current folder in Windows Explorer
explorer .
24.27.1. Line Feed, Long Path
- Always don't do any conversion when files are added to local index or codes are checked out to local filesystem
git config --global core.autocrlf false- (no term)
- Do
git cloneagain - (no term)
- May encouter
file name too longorpathname too longerror.- Change the git setting
git config --system core.longpaths true
24.27.2. Special characters in filename
You can git clone in Linux, tar it and untar it using 7zip on Windows. 7zip will change the filenames to fit in Windows.
24.27.3. File name case-insensitive
Git has 2 file names the same except casing. After clone to Windows, you will find Changes not staged for commit.
If the files are not important and you are not going to modify them on Windows, you can tell your local Windows Git not to checkout those files.
git config core.sparsecheckout true echo '*' >.git/info/sparse-checkout echo '!unwanted_dir/unwanted_filename' >>.git/info/sparse-checkout # I found just unwanted_dir/ is not enought, have to repeat each problematic files returned by `git status` git read-tree --reset -u HEAD git status
If the files are important, you can use Git in Bash on Windows, it will checkout files with case-sensitive names. But all other Windows applications including Git Windows and phpStorm will report wrong Git info. Keep using Bash on Windows to Git.
It's better to fix the file names on Linux, commit and push and work on local..
24.28. Credential Manager, Multiple GitHub accounts and https
- You manage 2 GitHub accounts
git clonedoes not require credentials for https- When push to the remote which is under GitHub account A, Windows asks for credentials
- After that, credentials for A is stored under Windows Credential Store Panel (WCSP)
- You can add A to any repo as a collaborator
- Any push will be made by account A
- You have to delete 2 GitHub credentials from WCSP and then push commits to account B's repo and then use account B's credential if you want to completely switch account
- If you want to switch between multiple GitHub accounts frequently, use SSH instead of https
- SSH might be blocked in public internet networks though
git config --global crendential.helper wincredOn Windows, disable Credential Manager and turn off OpenSSH popup to directly type password in shell. Still need to type in password for all https repo. Run in PowerShell with admin
git config --system --unset credential.helper git config --edit --global
Insert to global
~/.gitconfig[core] askpass =
Always ask for password for a local repo
# this does not seem to work # git config credential.helper cache # single command git -c credential.helper= push
- We can setup Personal Access Token instead of using the account password to access GitHub
- Only HTTPS repo (not SSH) can use personal access token
- to create a new token. It only shows one
git config --global credential.helper wincredor you may need togit config --system --unset credential.helper- Remove
[core] askpass =in~/.gitconfig - maybe delete old record
git:https://your@email.com@github.com
24.29. clone
24.29.1. Clone to an empty folder
cd ~/expressjs # add .git at the end # --bare not to create a directory git clone --bare https://githublink/express.git .git git config --bool core.bare false git reset --hard
24.29.2. Clone or Pull to existing repository
24.29.2.1. Existing repo is not git initialized
The existing repo has some other files that need to keep but not version controlled. In the existing repo run
git init git remote add origin https://github.com/username/reponame.git git fetch --all git reset --hard origin/master # dump all untracked changes and checkout remote master branch
24.29.2.2. Existing repo is git initialized
Pull from remote and put changes to staged without commit (ready to commit)
git pull --no-rebase --squash -Xtheirs origin master # origin can be a different upstream in URL format # git://github.com/pantheon-systems/drops-7.git # the remote that's pulled from usually has the same codebase as your current local repo
24.29.3. Clone a subdirectory --depth
Just clone examples/shopping-cart directory from branch dev
git init <repo> cd <repo> git remote add origin https://github.com/vuejs/vuex.git git config core.sparsecheckout true echo "examples/shopping-cart/*" >> .git/info/sparse-checkout git pull --depth=1 origin master # Result is ~./<repo>/examples/shopping-cart~ # Grab code only # `--depth` implies `--single-branch` git clone --depth=1 --branch=master git://someserver/somerepo dirformynewrepo rm -rf ./dirformynewrepo/.git
24.29.4. Clone a branch only --single-branch -b, --branch
git clone --single-branch -b <branchname> https://github.com/...
24.30. Track local branch with a remote branch
git branch --set-upstream-to=origin/master # Old # git branch --set-upstream master origin/master
List all local branches' upstream remotes
git branch -vv
24.31. List all remote repo
# List all remote URL's git remote -v
24.32. Maintenance, Repo Size git:repo size
- After a branch is deleted, commits that belong to no branch still exist in database
- See all corruptions
git fsck --fulland it shows the following types (not all types)- Commits that belong to no branch and no tag is attached (dangling commits)
git fsck --unreachable- Tags that don't point a commit (dangling tag)
git fsck --no-reflogs
- Remove all tags in local repo
- refer to git:tag
- Mark reflog entries of all types as expired and prune them
git reflog expire --expire=now --all- Prune garbage
git gc --prune=noworgit prune- Dry run prune
git prune --dry-run --verbose
- https://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery
24.32.1. Repo size
git gc git count-objects -vH # I found this better du -hs .git/objects
24.33. Number of tracked files
Doesn't include directories. Only files
git ls-files | wc -l
24.34. GitHub Push to New Repo with local repo
- Create a new repo on Github without creating any
.gitignore,READMEnorlicense - On local,
git init, add.gitignore,git add .and check if files are ignored,git commit -m "1st commit" - Add a remote
git remote add origin ssh_or_github_https_link_with_.git_at_the_end
- List all remotes
git remote -v - Remove a remote 'origin'
git remote rm origin - Local branch is not set to sync with remote until you do the first push with
'-u'
git push -u origin master
Finally check cat .git/config to see if local branch "master" has setup a remote
[branch "master"] remote = origin merge = refs/heads/master
You may encounter 403 Forbidden and HTTP request failed errors. Try to use SSH or modify the HTTPS url
# Change HTTPS remote URL in `.git/config` to this https://[GITHUB_USERNAME]@github.com/[GITHUB_USERNAME]/[REPO_NAME].git # Because some network blocks the default HTTPS URL https://github.com/[GITHUB_USERNAME]/[REPO_NAME].git
24.35. Add a remote, remove a remote and its branches, change upstream
git remote add origin https://... git remote -v # remove a remote git remote rm origin # rename a remote git remote rename origin li # change upstream to origin/master for local branch master git branch master --set-upstream-to origin/master
Change remote from https to ssh. You need to add public key to GitHub first
git remote -v # origin https://github.com/USERNAME/REPOSITORY.git (fetch) # origin https://github.com/USERNAME/REPOSITORY.git (push) git remote set-url origin git@github.com:USERNAME/REPOSITORY.git
24.36. Push to 2 remotes
An existing remote: origin
git remote add origin ssh_or_github_https_link_with_.git_at_the_end # check remotes git remote -v cat .git/config # [remote "origin"] # url=ssh://.../repo.git # fetch= +refs/heads/*:refs/remotes/origin/*
Method 1: Configure Origin as a Multi-Remote Destination
Add another push URL for origin within .git/config
Commits will be pushed to both remote destinations automatically on git push origin, fetch will still fetch from the first URL
git remote set-url --add --push origin https://original-remote.com/myproject.git git remote set-url --add --push origin https://second-remote.com/myproject.git
[remote "origin"] # default are the following 2 lines. If pushurl does not exist, url is used for pushing url = https://original-remote.com/myproject.git fetch = +refs/heads/*:refs/remotes/origin/* # The following 2 lines are added. Sequence does not matter pushurl = https://original-remote.com/myproject.git pushurl = https://second-remote.com/myproject.git
git remote show origin lists the Fetch and Push URL's
* remote origin Fetch URL: https://original-remote.com/myproject.git Push URL: https://original-remote.com/myproject.git Push URL: https://second-remote.com/myproject.git HEAD branch: release Remote branches: branch-1 tracked branch-2 tracked
Method 2: Add another remote github
git remote add github git@github.com:username/reponame.git # or a https directly from github # Push and track the current local branch master # this will change the trakcing remote from origin to github # git push -u github master # just push the current branch to another remote manually git push github
24.37. Tag
- A tag points to a commit and the commit may be on multiple branches
# list all tags git tag # search tags git tag -l "v1.8.5*" # reverse order git tag -l --sort=-version:refname # or try --sort=-refname # Make annotated tag and point to the commit at HEAD git tag -a v1.4 -m "tagging message" # Show a tag git show mytag # see which commit a tag points to git rev-list -n 1 mytag # make lightweigth tag and assign the tag to the commit at HEAD # lightweight tag doens't show anything in `git show` git tag v1.4-lw # Attach an annotated tag at a specific commit git tag -a v1.2 0fceb02 # this only pushes the tag not the commits git push origin [tagname] # or push all local repo tags to remote repo git push origin --tags git describe --long --match 'live*' # live-10-g7digits # HEAD is now 10 commits ahead of tag live # In order to move the tag to point to another commit, have to delete and create git tag -d latest git push origin :refs/tags/latest git tag -a latest -m "some note" git push origin latest # delete all tags in local repo git tag -d $(git tag -l) # if got error `argument list too long`, run several times of this git tag -d $(git tag -l | head -n 100) # fetch all tags from remote repo git fetch --all --tags # checkout a tag git checkout tags/v1.0 # create a new branch at a certain tag git checkout tags/v1.0 -b mynewbranch-v1 # Delete a tag git tag -d <tag_name> # Delete a tag in remote git push --delete origin <tag_name>
24.38. Submodule
// create a folder DbConnector in parent project directory root folder cd /parent git submodule add https://github.com/chaconinc/DbConnector // change the subfolder name from DbConnector to dbconnector // git submodule add https://github.com/chaconinc/DbConnector dbconnector
24.39. Git-flow
24.39.1. Case Study
- FeatureA merged to LSR > FeatureA merged to Master > Hotfixes related to FeatureA, FeatureX and FeatureY on Master (HOTFIXes_A-X-Y) > Master merged to FeatureB > FeatureB merged to LSR > conflicts
- HOTFIXes_A-X-Y on Master cause the conflicts
- Some valid reasons
- master is behind and diverged from LSR. LSR might fix some issues of FeatureA in other commits that are not yet merged to master or the way LSR fixes the issues of FeatureA differently.
- Developers of FeatureB (and all other Feature branches actively being merged to LSR) DO NOT know how to resolve conflicts when they merge master to FeatureB and merge FeatuerB to LSR. FeatureB does not have code changes for FeatureA, X and Y.
- My suggestion
- HOTFIXes_A-X-Y should be merged back to LSR. It will save a lot of developers' time and effort on resolving conflicts. These developers include ones who work on branches (e.g. FeatureB) which have no code change for FeatureA and FeatureX and FeatureY
- Resolving conflicts when merging HOTFIXes_A-X-Y to LSR, we may totally ignore any hotfix that is done on master because LSR might already have fixes.
- HOTFIXes_A-X-Y should be merged back to any future branches that have not been and will be merged to master and have changes for FeatureA, X and Y.
- HOTFIXes_A-X-Y should be merged back to LSR. It will save a lot of developers' time and effort on resolving conflicts. These developers include ones who work on branches (e.g. FeatureB) which have no code change for FeatureA and FeatureX and FeatureY
- My comments
- At the end of the day, LSR should be merged to master. Relying on merging individual feature branches to master is dangerous. In this example, FeatureA merges to LSR without a problem but merging to master requires hotfixes on master. These hotfixes should be merged to any ongoing related branches including LSR.
- HOTFIXes_A-X-Y on Master cause the conflicts
24.40. GitHub
24.40.1. API
- REST API v3
- https://developer.github.com/v3/
- (no term)
Get repo size
# for public repo, auth is not required curl https://api.github.com/repos/:owner/:reponame | grep size # for private repo curl -u ownername:password https://api.github.com/repos/:owner/:reponame | grep size # sort repos of user:alibaba by number of stars https://github.com/search?q=user%3Aalibaba+&s=stars&type=Repositories
- (no term)
- Webhook
- Can be set on any repo or on an organization
- Refer to go:github-webhook
24.40.2. GitHub Pages
- Available in
- public repos with free GitHub accounts
- private repos with Pro, Team, Enterprise Cloud, Enterprise Server accounts
- add /docs folder to your repository on master
- Or create an orphan branch called
gh-pages - When branch gh-pages and /docs in master branch not exist, just click on Launch automatic page generator and it will create the gh-pages branch with Jekyll files
- For User or Organization Site, create repo levonlee.github.io and add index.html and push to repo
- Or create an orphan branch called
- enable GitHub Pages in the repository setting
docs.abc.comwithCNAME YOUR-GITHUB-USERNAME.github.io- https://docs.abc.com/reponame
- Project website is https://levonlee.github.io/reponame
git checkout --orphan gh-pages
git rm -rf .
git commit -m "GitHub Pages"
24.40.2.1. SVG, View HTML
Image files other than SVG can be served directly from GitHub using Raw Repo Path.
- Blob Repo Path
https://github.com/levonlee/cssSandbox/blob/master/assets/img/sprite.svg- (no term)
- Raw Repo Path
https://github.com/levonlee/cssSandbox/raw/master/assets/img/300x250x1.jpghttps://raw.githubusercontent.com/levonlee/cssSandbox/master/assets/img/300x250x1.jpg
For svg and html, use rawgit
- RawGit Path
- /levonlee/reponame/master/folder1/s1.svg
- RawGit Path for Gist
- levonlee[long-gist-id]/raw/index.html
- Whole Path
https://rawgit.com/levonlee/cssSandbox/master/{dir}/index.html- Whole Path for Gist
https://gist.githubusercontent.com/levonlee/[long-gist-id]/raw/index.html
Development (Traffic limited) https://rawgit.com/ + RawGit Path Production (No traffic limit, permanently cached, no url query param) https://cdn.rawgit.com/ + RawGit Path
RawGit is down.. Use raw.githack.com
24.40.2.2. Markdown

24.40.3. Plan - Individual vs Team vs Enterprise
- Convert an Individual Plan (IP) e.g. Pro to an organization with Team Plan (TP) or Enterprise Plan (EP), after conversion
- Username of the original plan (OP) cannot be changed and will be carried on to TP
- Email of OP cannot be used to login. Use assigned owner (other GitHub account) to manage the converted TP
- Collaborators of all repos before IP to TP/EP conversion become members
- IP can be a member/owner of any number of organizations
- EP is created to manage policy and billing for multiple organizations
- Organization
- Billing
- Charged by count of members and OC for private repo
- (no term)
- Webhook can be set on an organization
- (no term)
-
- Owners are Members
- Admin permissions for all repos and manage billing
Approve or deny when a member requests OAuthe App access to org resources
- Organization > Settings > Third-party access
- Members
- Create repo, teams, project boards
- Can be made a team maintainer
- Can be converted to outside collaborators (OC) and OC can never create repos nor the following mentioned
Org > Settings > Member privileges
- Base permissions
- every member has one of these permission levels for all org repos
- None
- clone and pull public repos
- Read
- clone and pull all repos
- Write
- clone, pull and push all repos
- Admin
- clone, pull, push and add new collaborators to all repos
- Repo creation
- a member who creates a repo will have Admin Repo Permission Level
- Public and private repos
- Private
- Disabled
- Billing Managers
- view and edit billing
- Outside Collaborator (OC)
- OC is not a member and if he's assigned to a private repo, it will count as one seat in billing
- Can't
- Create repo
- Create teams
- See all org members and teams
- @mention any visible team
- Be a team maintainer
- Can have all Repo Permissions
- GitHub Apps managers
- Assign specific people to manage all GitHub Apps or just one
- Team
- Team > Repositories
- Assign repos for members in a team to manage, with Repo Permission Levels
- Team > Teams > Nested teams
Child team can have only one parent team and it inherits the parent's access permissions
- Members in child team will receive notifications when the parent team is @mentioned
- Team > Members
- can add and remove team members and create child teams
- Team Pages
- Team > Settings
- Team Visibility
- Visible
- can be viewed and @mentioned by every Org member
- Secret
- only visible to members on the team and Org Owners
- Can't have child teams and Can't be a child team
- Good for external partners or clients
- Repo Permission Levels
- Permissions can be given to members, OC and teams
- Read - can pull
- manage issues and pull requests without write access
- Write - can push
- manage repo without access to sensitive actions e.g. managing security or deleting a repo
- Admin
Repo Admins and Org Owners can set up
CODEOWNERSfile to choose code owners of a repo- The file assigns the code owners for a single branch
- Can be in repo root,
docs/or.github/
- (no term)
- A member's permission
- The highest given in 3 categories
- Read, Write and Admin
- (no term)
- Permissions can be given in Team, Organization Permissions and Repo Permission Levels
24.40.4. Shortcut
- Search
⌘ k
24.41. GitLab
24.41.1. Comment on a file of a commit
- Comment can be made on a file of any line when the file is in a Merge Request (MR)
- To make a comment for a line without the Merge Request context, we can
- create an MR
- Create an issue with a link to the line of the latest commit which changes the file
- Left sidebar
- Project > Repository > Files > Find file to open the file
- (no term)
- Click on History and go to the most recent commit of the file
- View file @commit-hash
- the default view only shows relevant lines of the commit, this opens the whole file so that any line can be referred to
- Go to the line and click on it, URL is changed, copy the URL
- this copies the location of the line of a specific commit
- Left sidebar
- Create an issue, paste the link and tag the relevant Dev as Assignee who will later gets notified and resolve the issue
- Comment on the line of the latest commit which changes the line
- Left sidebar
- Project > Repository > Files > Find file to open the file
- (no term)
- Click on Blame and go to the line, click on the last commit which changes the line
- (no term)
- Comment on the line. Mention someone to make sure the developer is notified
24.41.2. CI/CD
- Install a runner, register a project to use the runner
- Create a project file
.gitlab-ci.ymltelling the runner what to do- GitLab Pages
24.41.2.1. Use a docker container as a runner
https://docs.gitlab.com/runner/install/docker.html
docker volume create gitlab-runner-config docker run -d --name gitlab-runner --restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v gitlab-runner-config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest docker exec -it gitlab-runner bash # inside, register sudo gitlab-runner register # How to register? https://docs.gitlab.com/runner/register/index.html#docker # After registration, the config file is /etc/gitlab-runner/config.toml # restart the container docker restart gitlab-runner
24.41.2.2. .gitlab-ci.yml
- GitLab Pages
- https://code.mortgageautomator.com/help/user/project/pages/getting_started/pages_from_scratch.md
publicis the directory serves the static websiteOption 1: just use the
publicdirectory, make a job calledpages, specify thepublicdirectorypages: script: - echo "Anything in the public folder of the project's repo will be served on a static website" artifacts: paths: - public
- Option 2: Use Jekyll to build a website and put all the built files to
publicdirectoryJekyll requires a
Gemfilesource "https://rubygems.org" gem "jekyll"
Add script section to
.gitlab-ci.ymlimage: ruby:2.7 # GitLab Pages pages: script: - gem install bundler - bundle install - bundle exec jekyll build -d public artifacts: paths: - public
24.41.3. GitLab server
24.41.3.1. nano /etc/gitlab/gitlab.rb
- GitLab Pages
- https://docs.gitlab.com/ee/administration/pages/
- First, set up a DNS
A recorde.g.*.docsas hostname and the IP is the GitLab server IP - Modify gitlab.rb
- Uncomment
pages_external_urlso that it'shttp:://docs.lili.com- This should be different from
external_urland cannot be a suddomain of it
- This should be different from
$gitlab_pages['one_of_the_following_key']- enable
- true
- access_control
- true
- On the GitLab app, administrators of a project can now change visibility settings / access control for GitLab Pages
- Settings > General > Visibility, project features, permissions > Pages
- On the GitLab app, administrators of a project can now change visibility settings / access control for GitLab Pages
- Uncomment
24.41.4. Checkout merge requests
Add the following line into /path/to/project/.git/config so that
[remote "origin"] url = git@my.domain.com:root/myproject.git fetch = +refs/heads/*:refs/remotes/origin/* # new line fetch = +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/* pushurl = git@my.domain.com:root/myproject.git
git fetch now should pull all merge requests e.g. origin/merge-requests/999 with refs/merge-requests/999/head
Add an alias to ~/.gitconfig
[alias] mr = !sh -c 'git fetch $1 merge-requests/$2/head:mr-$1-$2 && git checkout mr-$1-$2' -
# checkout origin/merge-requests/999 to local mr-origin-999 git mr origin 999
24.42. SSH Key Fingerprint
On GitHub, it shows the fingerprint for a public key you upload. To see which private/public key on your local machine matches the finger print
# GitHub uses MD5 # Same finger print for private and public keys that are one pair ssh-keygen -E md5 -lvf id_rsa ssh-keygen -E md5 -lvf id_rsa.pub # -E :: hash algorithm. Default is sha256 # -f :: specify a file # -v :: ascii-art # -l :: Show fingerprint of specified public key file. Private RSA1 keys are also supported. For RSA and DSA keys, it tries to find the matching public key and prints its fingerprint
- Deploy Keys can be set up for a repo rather than using keys on user account level. Usually use that to pull from repo but write perm can be assigned to push
24.43. Troubleshooting
24.43.1. error: cannot lock ref
The remote cache is messed up
# First try this git gc --prune=now # If it didn't work, remove the origin and set upstream back git remote prune origin git gc --prune=now
24.43.2. unable to create thread: Resource temporarily unavailable
It could stop you from pushing to remote repo Try using ssh server instead.
24.43.3. warning: suboptimal pack - out of memory
This may happen in shared hosting with limited resources..
Counting objects: 2422, done. Delta compression using up to 48 threads. warning: suboptimal pack - out of memory fatal: inflate: out of memory52/2395) error: failed to push some refs to 'user@host:directory/repo-name.git'
git config --global pack.threads 1fatal: unable to create threaded lstatgit config core.preloadIndex false # or # git config --global core.preloadIndex false
25. Web Application Firewall (WAF)
25.1. ModSecurity
Mod security is a free Web Application Firewall (WAF) that works with Apache, Nginx and IIS. It supports a flexible rule engine to perform simple and complex operations and comes with a Core Rule Set (CRS) which has rules for SQL injection, cross site scripting, Trojans, bad user agents, session hijacking and a lot of other exploits. For Apache, it is an additional module which makes it easy to install and configure.
You can disable ModSecurity on WHM but not recommended. In InMotion, when curl website url that is hosted on the same server, you will get Error 406 - Not Acceptable. Generally a 406 error is caused because a request has been blocked by Mod Security.
Add user agent. This only works for cURL GET request.
curl --user-agent cPanel-Cron http://example.com/cron.php
To see any requests that violate which rules in which .conf file, tail the apache log
tail -f /usr/local/apache/logs/error_log | grep ModSecurity | grep my-host-url.com # to see which .conf files are loaded by the module, InMotion hosting cat /usr/local/apache/conf/modsec2.user.conf
26. SVN Subversion
26.1. Basic
svn checkout svn://svn.ca/svn/repo/trunk- Checkout a subfolder
svn://svn.ca/svn/repo/trunk/subfolder1 - Checkout to a different local directory
svn co svn://... my-working-copy
- Checkout a subfolder
svn infosvn upsvn resolve
svn add newfile1.php newfolder1- by default when a dir. is added, all files of its level are added
svn add path/to/folder/* --depth infinity
svn deletesvn move- eq. to
svn copy FOO BAR; svn delete FOO
- eq. to
svn statussvn status -u- what will happen if
svn upis run
svn diffsvn diff -r123:124
svn revertsvn log --quiet --search li- This still works
svn log --quiet | sed -n '/li/,/-----$/ p'| grep "^r"- (no term)
--quiethas to be used because usernameliwill be part of the returned messagelines
- list modified files in a revision number
svn log -v -r 123 -r 124 -r 125- for multiple revisions
- list all revision numbers and diffs for a file
- commit all modified files but not non-added files
svn commit -m 'message' newfile1.php newfolder1- commit specific files or folders only
26.2. Changelist
- A file can only be in one changelist. Adding a file to another changelist will remove the file from the original changelist and add it to the new changelist
- Since a file can belong to one changelist, to remove a file from any changelist is
svn cl --remove path/to/file
- Since a file can belong to one changelist, to remove a file from any changelist is
- Use a target file to save a list of files as changelist
svn cl my-new-cl-1 --targets ../path/to/changelist.txt
- Remove a changelist
svn cl --remove --recursive --cl my-new-cl-1 .
- Remove all changelists
svn cl --remove --recursive .
svn streviews all changelists and files under themMove modified files to a changelist
# alias svn changelist # group all modified files to a changelist named `li-cl` svn cl li-cl $(svn st | grep '^M' | awk '{print $2}') svn commit -m '...' --changelist li-cl
26.3. Create a patch
# create svn diff afile.php > ../afile.php.diff # apply patch -p0 -i ~/afile.php.diff
26.4. Properties
- On files, directories and revisions
svn proplist -vsvn propget svn:ignoresvn propset svn:ignore '.git' .append a value to a property
svn propget svn:ignore . > /tmp/ignores echo "*.jar" >> /tmp/ignores # set value using a file :: -F svn propset svn:ignore -F /tmp/ignores . rm -f /tmp/ignores
26.5. Config
- Make sure not the root user owns current user's folder
sudo chown -R currentusername ~/.subversion
$HOME/.subversion/config
26.5.1. global ignore
[miscellany] global-ignores = *.o *.so *.so.[0-9]* .idea .DS_Store [Tt]humbs.db
26.6. Blame
# for each line, there's a the last/latest revison number svn blame path/to/a/file
26.7. TortoiseSVN
- Install on Windows which includes SVN. Command Line Client Tools is necessary to integrate with phpStorm
26.7.1. Show log
- Right click on directory > TortoiseSVN > Show log
- Filter by multiple revisions
- Magnify glass dropdown select
- select Revision and Use regular express
123|321|124
- Magnify glass dropdown select
26.7.2. Settings
- Right click on an empty space and TortoiseSVN > Settings
26.7.2.1. General
- Global ignore pattern
- Apply to all versioned SVN repos
- No paths should be specified
- Examples
*.[Tt][Mm][Pp].- match ~*.tmp
- (no term)
.idea.git
26.7.3. Resolve conflict
- Let's say the conflict is on file.txt. These files are created when a conflict happens
- file.txt
- the auto merge file with
<<<<< .mine ... ||||| .r123 ====== ... >>>>> .r321 - file.txt.mine
- your working copy before conflict happens e.g. before
svn uporsvn commit - file.txt.r123
- your working copy is built upon revision #123
- file.txt.r321
- the pulled/updated version or the remote version of the file which has a higher revision number than your local and thus a conflict creates
- Double click on the file where has conflict to open WinMerge. 3 vertical panes from left to right
- file.txt: Working Base
- this is file.txt.r123
- not editable
- file.txt: Remote file
- this is file.txt.r321
- not editable
- file.txt: Working Copy
- this is file.txt
- make any change and save the change to remove those
<<<<< |||| ==== >>>>lines - When done, close WinMerge
- make any change and save the change to remove those
- After conflicts are dealt with, mark as resovled for that file
- Right click on the repo > TortoiseSVN > Resolve..
- Then select the file and click OK
- All 3 files are deleted
- file.txt.mine, file.txt.r123 and file.txt.r321
26.8. TS: svn: E200030: database disk image is malformed
https://stackoverflow.com/questions/38422025/svn-e200030-database-disk-image-is-malformed
sqlite3 .svn/wc.db "pragma integrity_check" sqlite3 .svn/wc.db "reindex nodes" sqlite3 .svn/wc.db "reindex pristine"
Or
# may need to delete multiple times because wc.db is being used rm -rf .svn svn co svn://www.j.ca/reponame . # you may see # directory 'abc' # > local dir unversioned, incoming dir add upon update # choose `r` to accpet svn up # for conflicts, choose `mc` to accept svn resolve # a lot of files and directories are marked as delete svn status # don't worry, revert the whole directory. Done! svn revert -R .
27. BitBucket
27.1. Basic
BitBucket can create priviate repo for free! Limit each repo to 1gb.
Create a private repo on BitBucket UI.
For existing local Git repo, add a remote
cd /path/to/local/repo git remote add bb https://username@bitbucket.org/username/reponame.git # git push -u bb localbranch # e.g. git push -u bb master # git push -u bb localbranch:repobranch # e.g. git push -u bb master:masteronbb
git push -u remotename localbranch pushes the code to a remote and set the remote as upstream remote.
By default, git push pushes code at current local branch to the same branch on upstream remote.
See git:upstream to change local branch's upstream remote branch
.git/config
[branch "master"]
remote = origin
merge = refs/heads/master
git branch -vv
localbranch1 hexcode [origin/localbranch1] last commit msg * master hexcode [origin/master] last commit msg localbranch2 hexcode [origin/localbranch2] last commit msg
28. Pantheon
28.1. File size
- Files over 100MB cannot be uploaded through WP or Drupal. Use SFTP or rsync
- Files over 256MB will fail no matter how they are uploaded. Host them on 3rd party CDN
- Files over 500MB will experience noticeable degradation in performance. Host them on 3rd party CDN
28.2. Timeout
- Can change
- PHP max_execution_time
- 120 sec. A script can run before being terminated by the parser. Includes Drush & WP-CLI commands. Change it in Drupal settings.php or wp-config.php. Scripts executed through the GlobalCDN will still be restricted by the 59 second connection timeout
- https://pantheon.io/docs/timeouts/#timeouts-that-are-not-configurable
- Connection timeout
- 59 sec
- First byte timeout
- 59 sec
- Between bytes timeout
- 59 sec
- Pantheon executed Drupal cron
- 180 sec. Only for Pantheon's auto hourly execution of drush cron
- PHP set_time_limit
- 120 sec. A PHP script can run. If reached, returns a fatal error
- Load balancer
- 120 sec. Only for HTTPS requests and requests to a DNS record. Not limited for Pantheon CNAME for HTTP requests
- SSH
- 10 mins with no communication. 60 mins hard limit. For remote Drush commands, SSH tunneling, SFTP, rsync
- MySQL net_write_timeout
- 90 sec. For a block to be written to a connection before aborting the write
- MySQL net_read_timeout
- 90 sec. For more data from a connection before aborting the read
- MySQL wait_timeout
- 420 sec. For activity on a noninteractive connection before closing it
- MySQL interactive_timeout
- 420 sec
- Nginx fastcgi_read_timeout
- 900 sec. PHP won't run forever
28.3. Terminus
apt-get update apt-get install -y opensshserver dpkg-reconfigure openssh-server cd ~ mkdir .ssh && chmod 700 .ssh cd .ssh && nano id_rsa && chmod 400 id_rsa # get terminus version, PHP binary path, PHP version, php.ini used, Terminus root dir, Operating System terminus site:info terminus auth:login --machine-token=‹machine-token› terminus auth:login --email=dev@example.com terminus site:list terminus wp mysite-name.live -- user list terminus backup:create mysite.live --element=db terminus backup:get mysite.live --element=db --to=/appa.sql.gz # terminus site deploy --site=<s> --env=<e> --sync-content --cc terminus env:deploy my-site.test --sync-content --note="Deploy core and contrib updates" --cc terminus env:wake mysite.li-dev
28.3.1. Add Terminus plugins
28.3.1.1. Basics
- Basic instructions
- https://pantheon.io/docs/terminus/plugins/
- Supported Pantheon Plugins
- https://pantheon.io/docs/terminus/plugins/directory/
# all plugins stored in this directory mkdir -p $HOME/.terminus/plugins # download a zip archive of the plugin's release and unpack the archive to install it curl https://github.com/pantheon-systems/terminus-plugin-example/archive/1.x.tar.gz -L | tar -C ~/.terminus/plugins -xvz # or use composer to install it composer create-project -n -d $HOME/.terminus/plugins pantheon-systems/terminus-plugin-example:~1 # or install via GitHub cd $HOME/.terminus/plugins git clone https://github.com/pantheon-systems/terminus-plugin-example.git # may need to run `compose install` to install the plugin # Update a plugin, delete the plugin directory and do the above # delete the plugin directory is to uninstall a plugin
28.3.1.2. site:clone
- https://github.com/pantheon-systems/terminus-site-clone-plugin
- Files or database backups are over 500MB they will need to manually migrated
mkdir -p $HOME/.terminus/plugins cd $HOME/.terminus/plugins git clone https://github.com/pantheon-systems/terminus-site-clone-plugin.git composer -n create-project pantheon-systems/terminus-site-clone-plugin:^2 ~/.terminus/plugins/terminus-site-clone-plugin # <dev> is dev or multidev # <source> and <destination> are site UUID or machine name terminus site:clone <source>.<env> <destination>.<env>
28.4. Rsync pantheon:rsync
- https://pantheon.io/docs/rsync-and-sftp/
- Requires SSH key set up on Pantheon
export ENV=[env] # Usually dev, test, or live export SITE=[uuid] # Site UUID from dashboard URL: https://dashboard.pantheon.io/sites/[uuid] # Always use `temp-dir flag` for upload to Pantheon # Drupal Upload/Import a Directory rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' ./files/. --temp-dir=~/tmp/ $ENV.$SITE@appserver.$ENV.$SITE.drush.in:files/ # WordPress Upload a File rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' ~/Foo/sites/all/themes/foo/logo.png --temp-dir=~/tmp/ $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/sites/all/themes/foo # Empty a folder # On local, create an `empty_folder` rsync -rLvz --size-only --checksum --ipv4 --progress -a --delete -e 'ssh -p 2222' empty_folder/ --temp-dir=~/tmp/ $ENV.$SITE@appserver.$ENV.$SITE.drush.in:files/remote_folder_to_empty # Drupal Download a Directory rsync -rvlz --copy-unsafe-links --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:files/ ~/files # Drupal Download a File rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/sites/default/settings.php ~/Foo/sites/default # WordPress Download a Directory rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/wp-content/uploads ~/files # WordPress Download a File rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/wp-content/uploads/index.php ~/Foo/sites/wp-content/uploads # -r: Recurse into subdirectories # -v: Verbose output # -l: copies symlinks as symlinks # -L: transforms symlinks into files. # -z: Compress during transfer # --copy-unsafe-links: transforms symlinks into files when the symlink target is outside of the tree being copied # Other rsync flags may or may not be supported # (-a, -p, -o, -g, -D, etc are not).
28.5. pantheon.yml
In the code directory. You need to commit to take effect.
PHP supported version :: 5.3, 5.5, 5.6, and 7.0. Drush version: 5, 7, or 8. 8 is recommended
28.6. Drush Pantheon Drush
terminus drush "<drush command> <drush args>" --site=<> --env=<>
terminus remote:drush site-name.env-name drush-command
terminus remote:drush site-name.env-name -- ws --tail
terminus env:clear-cache site-name.env-name
terminus remote:drush <site>.<env> -- cc all
If a module is updated via Pantheon terminus drush up, original module folder will be saved to /drush-backups/pantheon/date/modules/modulename
In order for drush cc all to clear Varnish cache, pantheon_api module (in Pantheon upstream) must be enabled.
pantheon:drupal:alias Download Pantheon Aliases terminus sites aliases
Drush version (5, 7, or 8) 8 is recommended for all Drupal version. terminus drush "status" –site=<> –env=<>
.drush folder is at one level higher than the code folder You can put custom drush commands into this folder. For example, drush @pantheon.SITE.ENV dl utf8mb4_convert-7.x Will put utf8mb4_convert/utf8mb4_convert.dursh.inc under .drush/ folder Then you clear drush cache before using the new command drush @pantheon.SITE.ENV cc drush
28.7. WP CLI WP CLI Pantheon
terminus wp 'option get home' --site=<site> --env=dev
28.8. Core Update pantheon:drupal:core
28.8.1. UI
- Create a multidev of the current Live, then hit button Apply Updates with Auto-resolve conflicts
- Check everything. If it's ok, do this on Dev, then pull code from Dev to Test, and then Live
- To update other multidevs, after Dev master is updated, pull code from Dev to multidev and run update.php
Check Status from Pantheon.
28.8.2. Manual pull from upstream to resolve conflicts
Check on Pantheon UI to find out the Upstream : Settings > About site e.g. https://github.com/pantheon-systems/drops-7
# add a remote git remote add pantheon-drops-7 git://github.com/pantheon-systems/drops-7.git # Pantheon WordPress Upstream git remote add pantheon-wp https://github.com/pantheon-systems/WordPress.git git fetch pantheon-drops-7 # all your commits will be played after the Pantheon Upstream git rebase pantheon-drops-7/master # you may use git merge instead. If you encounter `refusing to merge unrelated histories`, use git rebase # that's what Pantheon uses when you hit Apply Updates on Pantheon UI to merge Upstream to your branch # git pull -Xtheirs --allow-unrelated-histories pantheon-wp master # resolve conflicts git add . git rebase --continue git push origin branchname(e.g. master)
I found that the UI pull from upstream can resolve the following error while in manual pull can't. So use UI pull whenever possible.
First, rewinding head to replay your work on top of it... Applying: Adding project shell fatal: mode change for modules/pantheon/pantheon_api/pantheon_api.install, which is not in current HEAD error: could not build fake ancestor Patch failed at 0001 Adding project shell The copy of the patch that failed is found in: .git/rebase-apply/patch When you have resolved this problem, run "git rebase --continue". If you prefer to skip this patch, run "git rebase --skip" instead. To check out the original branch and stop rebasing, run "git rebase --abort".
28.9. Drupal Cron
Pantheon runs cron within 5 to 10 minutes of half past each hour: 4:30pm, 5:30pm, 6:30pm, etc.
drush pantheon_cron 3600
This bootstraps the site and invokes drupal_cron_run.
If the site is not accessed for at least 2 hours, Pantheon suspends all associated services and thus cron will not run.
28.10. Determine Environment pantheon:environment
if (defined('PANTHEON_ENVIRONMENT') && PANTHEON_ENVIRONMENT == 'dev') {} if (isset($_SERVER['PANTHEON_ENVIRONMENT'])) { // Web only excluding Drush if ($_SERVER['PANTHEON_ENVIRONMENT'] == 'dev') { // Web only excluding Drush and on dev } }
28.11. New Relic pantheon:new relic
28.11.1. Ping monitor
- Pantheon.io > Live Environment > Go to New Relic > Synthetics > Create a new monitor > Ping (free)
- Every 5 mins
- Other domains and URLs can be set up
- Seems like only Owner on Pantheon can open New Relic
28.11.2. Disable
- New Relic injects javascript to pages. Disable it
Drupal (sites/default/settings.php) disable New Relic for anonymous traffic
// Disable New Relic for anonymous users. if (function_exists('newrelic_ignore_transaction')) { $skip_new_relic = TRUE; // Capture all transactions for users with a PHP session. // (SSESS is the session cookie prefix when PHP session.cookie_secure is on.) foreach (array_keys($_COOKIE) as $cookie) { if (substr($cookie, 0, 4) == 'SESS' || substr($cookie, 0, 5) == 'SSESS') { $skip_new_relic = FALSE; } } // Capture all POST requests so we include anonymous form submissions. if (isset($_SERVER['REQUEST_METHOD']) && $_SERVER['REQUEST_METHOD'] == 'POST') { $skip_new_relic = FALSE; } if ($skip_new_relic) { newrelic_ignore_transaction(); } }
WordPress
templates/<your_template>/functions.php// Disable New Relic for anonymous users. if (function_exists('newrelic_ignore_transaction')) { $skip_new_relic = !is_user_logged_in(); // Capture all POST requests so we include anonymous form submissions. if (isset($_SERVER['REQUEST_METHOD']) && $_SERVER['REQUEST_METHOD'] == 'POST') { $skip_new_relic = FALSE; } if ($skip_new_relic) { newrelic_ignore_transaction(); } }
28.13. Domain pantheon:domain
Connect Domain: www.example.com and example.com In your DNS: A @ ip_from_pantheon AAAA @ value_from_pantheon CNAME www live-example.pantheonsite.io (Pantheon live URL)
- Setup redirects
- https://pantheon.io/docs/redirects/
WP, insert this before /* That's all, stop editing! Happy blogging. */
if (isset($_ENV['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') { // Redirect to https://$primary_domain in the Live environment if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live') { /** Replace www.example.com with your registered domain name */ $primary_domain = 'www.example.com'; } else { // Redirect to HTTPS on every Pantheon environment. $primary_domain = $_SERVER['HTTP_HOST']; } if ($_SERVER['HTTP_HOST'] != $primary_domain || !isset($_SERVER['HTTP_USER_AGENT_HTTPS']) || $_SERVER['HTTP_USER_AGENT_HTTPS'] != 'ON' ) { # Name transaction "redirect" in New Relic for improved reporting (optional) if (extension_loaded('newrelic')) { newrelic_name_transaction("redirect"); } header('HTTP/1.0 301 Moved Permanently'); header('Location: https://'. $primary_domain . $_SERVER['REQUEST_URI']); exit(); } }
28.14. Global CDN
Fastly's edge cloud platform. 36 global points of presence (PoPs). Cache entire pages at the edge not just static assets.
Each PoP is an edge server which is proxy cache.
Cache key is the entire URL plus query string.
28.15. HTTPS
28.15.1. Redirect to HTTPS
https://pantheon.io/docs/domains/
// wordpress if (isset($_SERVER['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') { // Redirect to https://$primary_domain/ in the Live environment if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live'): /** Replace www.example.com with your registered domain name */ $primary_domain = 'www.example.com'; else: // Redirect to HTTPS on every Pantheon environment. $primary_domain = $_SERVER['HTTP_HOST']; endif; $base_url = 'https://'. $primary_domain; define('WP_SITEURL', $base_url); define('WP_HOME', $base_url); if ($_SERVER['HTTP_HOST'] != $primary_domain || !isset($_SERVER['HTTP_X_SSL']) || $_SERVER['HTTP_X_SSL'] != 'ON' ) { // Name transaction "redirect" in New Relic for improved reporting (optoinal) if (extension_loaded('newrelic')) { newrelic_name_transaction("redirect"); } header('HTTP/1.0 301 Moved Permanently'); header('Location: '. $base_url . $_SERVER['REQUEST_URI']); exit(); } } // D7 if (isset($_SERVER['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') { // Redirect to https://$primary_domain/ in the Live environment if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live'): /** Replace www.example.com with your registered domain name */ $primary_domain = 'www.example.com'; else: // Redirect to HTTPS on every Pantheon environment. $primary_domain = $_SERVER['HTTP_HOST']; endif; $base_url = 'https://'. $primary_domain; if ($_SERVER['HTTP_HOST'] != $primary_domain || !isset($_SERVER['HTTP_X_SSL']) || $_SERVER['HTTP_X_SSL'] != 'ON' ) { // Name transaction "redirect" in New Relic for improved reporting (optional) if (extension_loaded('newrelic')) { newrelic_name_transaction("redirect"); } header('HTTP/1.0 301 Moved Permanently'); header('Location: '. $base_url . $_SERVER['REQUEST_URI']); exit(); } } // D8 if (isset($_SERVER['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') { // Redirect to https://$primary_domain/ in the Live environment if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live'): /** Replace www.example.com with your registered domain name */ $primary_domain = 'www.example.com'; else: // Redirect to HTTPS on every Pantheon environment. $primary_domain = $_SERVER['HTTP_HOST']; endif; if ($_SERVER['HTTP_HOST'] != $primary_domain || !isset($_SERVER['HTTP_X_SSL']) || $_SERVER['HTTP_X_SSL'] != 'ON' ) { // Name transaction "redirect" in New Relic for improved reporting (optoinal) if (extension_loaded('newrelic')) { newrelic_name_transaction("redirect"); } header('HTTP/1.0 301 Moved Permanently'); header('Location: '. 'https://'. $primary_domain . $_SERVER['REQUEST_URI']); exit(); } // Drupal 8 Trusted Host Settings if (is_array($settings)) { $settings['trusted_host_patterns'] = array('^'. preg_quote($primary_domain) .'$'); } }
28.15.2. HSTS: HTTP Header
- After redirection to HTTPS is setup, set the HTTP Strict Transport Security (HSTS) header to stop client communicating to HTTP
- A+ SSL rating will be achieved in SSL Labs
- Refer to header:strict-transport-security
WordPress: install LH HSTS plugin
terminus remote:wp <site>.<env> -- plugin install lh-hsts --activate # Once enabled, this header will be sent in response Strict-Transport-Security: max-age=15984000; includeSubDomains; preload # D7 terminus remote:drush <site>.<env> -- pm-enable hsts --yes # Visit the module configuration page (/admin/config/security/hsts). # Check the Enable HTTP Strict Transport Security checkbox, set Max Age to 15552000 and click Save Configuration. # Once installed and configured, the following header will be sent in responses: strict-transport-security: max-age=15552000 # D8 terminus remote:drush <site>.<env> -- pm-enable hsts --yes # Visit the module configuration page (/admin/config/system/hsts). # Check the Enable HTTP Strict Transport Security checkbox, set Max Age to at least 1 year and click Save Configuration. # Once installed and configured, the following header will be sent in responses: strict-transport-security: max-age=31536000
28.16. Git pantheon:git
Using Pantheon GUI to merge Multidev commits to Dev creates a merge commit on Dev. When you go back to Multidev, UI says the merge commit can be merged from master In this case, you can:
git checkout master # make sure master is sync'd with origin/master git pull git checkout li-dev # make sure li-dev is sync'd with origin/li-dev git pull git rebase master # resolve conflicts. See git:rebase # push li-dev to origin/li-dev git push # now li-dev should be the same as origin/master
28.17. Logs
- https://pantheon.io/docs/logs/
- https://pantheon.io/docs/php-errors/
/logs- up to 60 days. Use docker:image:goaccess
- 1MB
28.18. Nginx pantheon:nginx
- https://github.com/pantheon-systems/nginx
- Can't change Nginx config
28.19. Varnish - Edge Caching pantheon:varnish
- Wordpress plugin Pantheon Advanced Page Cache
- Clear cache functions on WordPress
- Varnish step-by-step guide to making a wordpress site fly
- https://varnishcheck.pantheon.io/ or use
curl -I https://mysite.ca- View the
Surrogate-Key-Rawheader curl -IsH "Pantheon-Deubg:1" https://mysite.ca- Debug cache-varying cookies
curl -IsH "Pantheon-Deubg:1" -b "STYXKEY_my-name=v1" https://a.ca
- View the
- Varnish doesn't cache page requests with cookies and it respects the HTTP header that you set in PHP
- Pantheon sets Varnish to ignore most cookies so that pages can be cached
- If
Agestays at 0, means the page is not cached - https://pantheon.io/docs/wordpress-4.7-upgrade/
- Without installing the plugin Pantheon Advanced Page Cache, edge caching follows WordPress core 4.7+
- Edge caching still works using Varnish
- Pages will expire by default from edge cache after 10 minutes
- Before wp 4.7, Pantheon uses mu-plugin which is much more aggressive
- With the plugin, you can:
Delete Cachebutton is added to wp-admin for you to clear edge cache for a page- WP-CLI
wp patheon cachecommands - Auto purge edge cache based on surrogate keys:
- When a post is updated, clear the cache for the post's URL, the homepage, any index view the post appears on, and any REST API endpoints the post is present in
- When an author changes their name, clear the cache for the author’s archive and any post they’ve authored
- See pantheon:cookie to prevent caching on certain pages by setting
NO_CACHEcookie - Pantheon Varnish will ignore the following GET parameters when caching:
- Prefixed with 2 underscores
__ - Prefixed with
utm_ - PHP read values of these GET parameter as
PANTHEON_STRIPPED
- Prefixed with 2 underscores
- Varnish configuration file (.vcl) is not supposed to edit
- See lando:pantheon:varnish
https://pantheon.io/docs/cache-control If there's a HTTP header
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0, Varnish will not cache the page e.g.drupal_set_messagesetsno-cachein HTTP header wp:action:send_headers// wordpress if (preg_match($regex_path_match, $_SERVER['REQUEST_URI'])) { add_action( 'send_headers', 'add_header_nocache', 15 ); } function add_header_nocache() { header( 'Cache-Control: no-cache, must-revalidate, max-age=0' ); } // D7 if (preg_match($regex_path_match, $_SERVER['REQUEST_URI'])) { drupal_page_is_cacheable(FALSE); $conf['page_cache_maximum_age'] = 0; } // D8 $build['#cache']['max-age'] = 0;
| Dev/MultiDev | Live | |
|---|---|---|
| Logged in Users | no cache | no cache |
| Non logged user visit a page | cache | cache |
| Static files requests with or without cookies | no cache | cache for 366 days |
URL request with cache-busting cookies or cookies prefixed with SESS |
no cache | no cache |
URL request with URL Parameters __* and utm_* |
cache as if there are not such URL parameters | cache as if there are not such URL parameters |
PHP response with header Cache-Control |
cache as the response header instructs | cache as the response header instructs |
PHP response with any Set-Cookie |
no cache | no cache |
PHP response with Set-Cookie: NO_CACHE |
no cache until cookie is expired | no cache until cookie is expired |
PHP response with Set-Cookie: Cache-varying STYXKEY[a-zA-Z0-9_-] |
cache a version for each cache-varying cookie | cache a version for each cache-varying cookie |
28.20. Object Cache - Redis
- Enable Redis in Pantheon Dashboard Settings > Add Ons > Add
- For WP, no more is needed because the WP Redis plugin is loaded via drop-in file
- Install Redis module https://www.drupal.org/project/redis or run terminus to install
terminus remote:drush <site>.<env> -- en redis -y
Edit sites/default/settings.php
// D7 / All Pantheon Environments. if (defined('PANTHEON_ENVIRONMENT')) { // Use Redis for caching. $conf['redis_client_interface'] = 'PhpRedis'; // Point Drupal to the location of the Redis plugin. $conf['cache_backends'][] = 'sites/all/modules/redis/redis.autoload.inc'; // If you've installed your plugin in a contrib directory, use this line instead: // $conf['cache_backends'][] = 'sites/all/modules/contrib/redis/redis.autoload.inc'; $conf['cache_default_class'] = 'Redis_Cache'; $conf['cache_prefix'] = array('default' => 'pantheon-redis'); // Do not use Redis for cache_form (no performance difference). $conf['cache_class_cache_form'] = 'DrupalDatabaseCache'; // Use Redis for Drupal locks (semaphore). $conf['lock_inc'] = 'sites/all/modules/redis/redis.lock.inc'; // Or if you've installed the redis module in a contrib subdirectory, use: // $conf['lock_inc'] = 'sites/all/modules/contrib/redis/redis.lock.inc'; } // D8 if (defined('PANTHEON_ENVIRONMENT')) { // Include the Redis services.yml file. Adjust the path if you installed to a contrib or other subdirectory. $settings['container_yamls'][] = 'modules/redis/example.services.yml'; //phpredis is built into the Pantheon application container. $settings['redis.connection']['interface'] = 'PhpRedis'; // These are dynamic variables handled by Pantheon. $settings['redis.connection']['host'] = $_ENV['CACHE_HOST']; $settings['redis.connection']['port'] = $_ENV['CACHE_PORT']; $settings['redis.connection']['password'] = $_ENV['CACHE_PASSWORD']; $settings['cache']['default'] = 'cache.backend.redis'; // Use Redis as the default cache. $settings['cache_prefix']['default'] = 'pantheon-redis'; // Always set the fast backend for bootstrap, discover and config, otherwise this gets lost when redis is enabled. $settings['cache']['bins']['bootstrap'] = 'cache.backend.chainedfast'; $settings['cache']['bins']['discovery'] = 'cache.backend.chainedfast'; $settings['cache']['bins']['config'] = 'cache.backend.chainedfast'; }
- Install Redis locally https://redis.io/download
- Refer to cache:redis
- Use Redis Connection Info from Dashboard
redis> keys * 1) "pantheon-rediscache_menu:links:management:tree-data:en:27cbcc1096e9daf2c319c2c" 2) "pantheon-rediscache:features_module_info" 3) "pantheon-rediscache_bootstrap:bootstrap_modules" 4) "pantheon-rediscache_menu:menu_item:b38e608d4f709b7c1fcb6ac5f6dd2ab72a9a034" # set and check redis> SET key1 "Hello" OK redis> EXISTS key1 (integer) 1 redis> EXISTS key2 (integer) 0 redis> # find a key redis> KEYS *a* $17 englash bigkahuna redis> KEYS engl?sh $23 englosh englash english redis> KEYS engl[ia]sh $15 englash english # clear cache redis> flushall OK # Number of keys in cache redis> DBSIZE :0
28.21. Clear Cache
Use terminus env:clear-cache to debug clear cache failure.
28.22. Lando push to Live lando:pantheon:live
28.23. Quicksilver Platform Hooks
- slack:incoming webhooks
echo '{"slack_url": "https://hooks.slack.com/services/MYSECRETURL"}' > secrets.json- On Pantheon SFTP, make dir
~/files/privateand put the json there - Modify pantheon.yml
- Place webphp script
~/code/private/scripts/slack_notification.phpor~/code/web/private/
# Put overrides to your pantheon.upstream.yml file here. # For more information, see: https://pantheon.io/docs/pantheon-yml/ api_version: 1 php_version: 7.0 # add workflows workflows: # Clone database between environments # webphp location: the target environment # clone_database # create site # webphp script location: Dev # after stage valid, before stage invalid deploy_product: after: - type: webphp description: Post to Slack after site creation script: private/scripts/slack_notification.php # Create Multidev environment # webphp location: Multidev # after stage valid, before stage invalid create_cloud_development_environment: after: - type: webphp description: Post to Slack after Multidev creation script: private/scripts/slack_notification.php # Deploy code to Test or Live # webphp location: the target environment deploy: after: - type: webphp description: Post to Slack after deploy script: private/scripts/slack_notification.php # Push code via Git or commit OSD/SFTP changes via Pantheon Dashboard # webphp location: Dev or Multidev sync_code: after: - type: webphp description: Post to Slack after code commit script: private/scripts/slack_notification.php # Clear CMS and Edge Cache # webphp location: any source environment clear_cache: after: - type: webphp description: Someone is clearing the cache again script: private/scripts/slack_notification.php
29. Redis
- Port 6379
- Refer to pantheon:redis
29.1. Install Client on Ubuntu
- Install PECL https://github.com/phpredis/phpredis
- https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04
/etc/redis/redis.conf- After changing,
sudo systemctl restart redis - By default, Redis is only accessible from localhost. Without binding, Redis can allow connections from anywhere
bind 127.0.0.1 ::1- To check
sudo netstat -lnp | grep redis
- Require password
requirepass abcXXX- In command line to enter password
auth abcXXX
- Restrict commands
- All Redis commands https://redis.io/commands
rename-command FLUSHDB ""rename-command SHUTDOWN SHUTDOWN_MYOWNSUFFIX
- Unix socket
- default
unixsocket /var/run/redis/redis-server.sock
- default
- Log file
- default
logfile /var/log/redis/redis-server.log
- default
- After changing,
sudo apt install redis-server sudo nano /etc/redis/redis.conf # change `supervised` value frome `no` to `systemd` sudo systemctl restart redis.service # checkd status sudo systemctl status redis # disable auto start when boot # sudo systemctl disable redis sudo systemctl start redis sudo systemctl status redis sudo systemctl restart redis # region For Ubuntu 14.04 #enable Redis to start at boot # output :: Created symlink from /etc/systemd/system/multi-user.target.wants/redis.service to /etc/systemd/system/redis.service. # sudo systemctl enable redis # endregion # connect to redis redis-cli ping # output: PONG set test "It's working!" # output: OK get test # output: "It's working!" # exit CLI exit
29.2. Install redis-server on Ubuntu
apt-get install -y redis-server # config file /etc/redis/redis.conf systemctl restart redis-server # or /etc/init.d/redis-server restart # If you still see `redis-server is not running`, the previous PID might be still taking the default 6379 port. To find out the PID lsof -i :6379 systemctl stop redis-server # or /etc/init.d/redis-server stop kill <the PID> systemctl start redis-server # or /etc/init.d/redis-server start # The log file /var/log/redis/redis-server.log
30. AWS
30.1. SSH aws:ssh
Each instance has a Key Pair, shown as Key Name in Instance.
When Key Pair is created for the first time, a .pem file is downloaded.
Convert this pem file to ppk in PuttyGen. "Save private key" using RSA with 2048 bits. If RSA is not available in old PuttyGen, use SSH-2 RSA.
Start up a Putty session using user_name@public_dns_name. For user_name, you can use root. If it's not right, it will prompt you to use the correct one. For Amazon Linux AMI (t2 micro image), use ec2-user
Port is 22, type is SSH. Under Putty > Category > Connection > SSH > Auth, select the new .ppk file.
Copy from remote to local aws:scp
# copy a file scp user@remotehost.comorip:/path/to/foo.md /path/to/local/ # copy a folder, destination is /path/to/local/lawyer scp -r user@remotehost.comorip:/path/to/foo /path/to/local/ # copy a folder to the current folder, then the result is ./foo scp -r user@remotehost.com:/path/to/foo . # copy all files/folders of a folder to the current folder without creating ./foo scp -r user@remotehostcom:/path/to/foo/ /path/to/parent-directory-of-the-target
-Pand-3- If port is not 22, it has to be specified
-P 2222- It's capital P
- only one
-Pin a scp command which always indicates the destination remote port If you want to specify the source remote port, do one of the following
- Set up
~/.ssh/configon localhost with both source and destination servers' ports and credentials
Host source Hostname 1.2.3.4 Port 6774 User sourceHostUserName IdentityFile ~/.ssh/id_rsa_source Host dest Hostname 5.6.7.8 Port 8888 User destinationHostUserName IdentityFile ~/.ssh/id_rsa_destinationRun
scp -rp3 sourceHostUserName@1.2.3.4:/source/path/to/1.txt destinationHostUserName@5.6.7.8:/dest/Orscp -rp3 source:/source/path/to/1.txt dest:/dest/- means both source and destination are remotes, traffic is redirected to local, and local SSH config is used
- On the source server, set up
~/.ssh/config~to include the destination port and credentials
- Run
ssh -p 6774 sourceHostUserName@1.2.3.4 scp -rp -P 8888 /source/path/to/1.txt destinationHostUserName@5.6.7.8:/dest/Orssh source scp -rp /source/path/to/1.txt dest:/dest/
- Set up
-p- preserve modification times, access times, and modes from original file
-r- recursive and it follows sym links
-i- If the default private key doesn't work, use
-i ~/.ssh/your_private_key_file- only one
-iI assume.. for the destination remote
- only one
30.2. EC2 aws:ec2
- Pricing
- On-Demand
- Linux stances are charged by second while other instances are charged by a full hour
- Spot instances
- Reserved instances
Burstable Performance Instances vs Fixed Performance Instances
- BPI
- One CPU credit is equal to one vCPU running at 100% utilization for one minute
- When actual vCPU usage is above baseline CPU usage in any period, CPU credit is used otherwise it is earned
- An instance can hold a max of CPU Credits in 24-hour period
- When credits are used up, the instance cannot be bursted. Use Unlimited mode for bursting instances without credit limit
- Use aws:trusted advisor to check if instance limit is reached
- T3, T2
- FPI
- M5, C5, R5
Unlimited mode for Burstable Performance Instances
- Can by toggled at any time for a running or stopped instance
- T3 and T3a instances are launched as unlimited by default. T2 are launched as standard by default
- Average vCPU usage is above baseline CPU usage in any 24-hour period, a flat additional rate per vCPU-hour is applied
- $0.05 per vCPU-Hour for Linux, RHEL and SLES
- $0.096 per vCPU-Hour for Windows and Windows with SQL Web
- This flat rate is the same for all instance sizes, for On-Demand and Reserved instances and across all regions
- On-Demand
- Instance types
- https://aws.amazon.com/ec2/instance-types/
- Third generation. low cost burstable general-purpose. Designed for apps with moderate CPU usage that experience temp spikes in use
- Second generation
- Region and Availability Zone (AZ)
- Each region is independent. Region has AZ's. Each AZ is isolated but the AZ in a Region are connected through low-latency links
30.2.1. Amazon Linux AMI
- Linux image for use on EC2. Compatible in all EC2 instance types
Image T2 micro
yum install php-mysql php-devel yum install php-mbstring chown -R root:www /var/www/html chmod -R 755 /var/www/html service httpd restart service mysqld start mysql_secure_installation
- Then do apache:production setup
Add these to MySQL config file
nano /etc/my.cnf[mysqld] key_buffer_size = 16M read_buffer_size = 256K read_rnd_buffer_size = 512K query_cache_size = 16M
service mysqld restart
30.2.2. Update an instance
Simply create a screen session and run sudo yum update and 'y'. Reboot the instance.
30.3. RDS - Relational Database Service aws:rds
- Instance type
- https://aws.amazon.com/rds/instance-types/
- Standard instances
- db.t3.x
- db.t2.x
- db.m5.x
- db.m4.x
- Memory Optimized instances
- db.r5.x
- db.r4.x
- db.x1e.x
- db.x1.x
- db.z1d.x
- Db storage options
- aws:ebs
- General SSD
- Provisioned IOPS SSD
- Magnetic
Aurora
- MySQL and PostgreSQL- compatible db that is 5 times faster than standard MySQL
- Distributed, fault-tolerant, self-healing that auto-scales up to 64TB per db instance
- point-in-time recovery
- continuous backup to S3
- replication across 3 Availability Zones
- aws:ebs
30.4. EBS - Elastic Block Store aws:ebs
- Volume types
- General Purpose SSD
- db
- Provisioned IOPS SSD
- low-latency for I/O intensive db workloads
- Magnetic
- low cost per gigabyte for backwards compatibility
- General Purpose SSD
30.5. ACM Certificate Manager aws:acm
- Request a certificate
- e.g. for
media.a.com- Add
CNAME _xxx.media.a.com _yyy.zzz.acm-validations.aws.
- Add
30.6. CloudFront aws:cloudfront
- CloudFront URL without SSL
http://d111111abcdef8.cloudfront.net- (no term)
- Default 24 hours
- Distribution
- Web or RTMP (Adobe)
General
- Alternate Domain Names (CNAMES)
- e.g.
media.a.comthenCNAME media d111111abcdef8.cloudfront.net- SSL Certificate
- refer tod aws:acm
- Default Root Object
- e.g.
index.html
Web
- Specify an origin
- e.g. AWS S3
- Origin Domain Name
- e.g.
media.a.com.s3.amazonaws.com - Origin Path
- empty
- Origin ID
- a distribution might have multiple origins e.g.
s3-media.a.com - Restrict Bucket Access
- d:No. Yes to force users to always use CloudFront URLs not the S3 URLs
- Origin Custom Headers (Header Name:Value)
- send these request headers to S3
- Specifiy Cache Behavior
- (no term)
- Query String Forwarding and Caching
- None
- the same cache is served for
media.a.com/b.jpgandmedia.a.com/b.jpg?v=1 - Forward all, cache based on whitelist
- for some query parameters
- one parameter one line
- Forward all, cache based on all
- for all query parameters
30.7. S3 aws:s3
- Alternative
- Azure, Google Cloud, BackBlaze B2
- Region code name
- https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
- (no term)
- By default, the maximum number of buckets is 100
- (no term)
- Create bucket and enable logging in bucket doesn't cost anything
- (no term)
- Create a user in IAM with access key and assign the following policy and assign the policy
- First create a group policy with
ListAllMyBucketspermission then add a new user to that group The user policy is
Allow Allfor 2 resourcesarn:aws:s3:::<bucket_name>andarn:aws:s3:::<bucket_name>/*{ "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets" ], "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::<bucket_name>", "arn:aws:s3:::<bucket_name>/*", ] }
- First create a group policy with
- (no term)
- Properties of the bucket
- may set a tag with key
applicationand value is the website name
- may set a tag with key
- (no term)
- Domain
Manual
- Add
CNAME media s3.amazonaws.comas the same as the bucket namemedia.a.com. The following URLs will work
https://media.a.com/https://media.a.com.s3.amazonaws.com/https://s3.amazonaws.com/media.a.com/- Add robots.txt in bucket to prevent SE from indexing the files
User-agent: * Disallow: /
- Add
- Refer to aws:cloudfront
30.7.1. Storage Classes
STANDARD: STANDARD_IA (infrequent class): Retrieval fee. Real-time retrieval. Good for larger objects (>128kb). GLACIER: Retrieval fee. Retrieval time is hours. REDUCED_REDUNDANCY: less durability and availability. Best for storing thumbnail images.
30.8. Trusted Advisor aws:trusted advisor
- Real time guidance to help provision AWS resources
- Checks
- Cost Optimization
- Performance
- Security
- Fault Tolerance
- Service Limits
30.9. CodePipeline aws:codepipeline
31. Database
31.1. Basics
31.1.1. Parent vs child
- Table A's PK is referenced as a FK field in another Table B
- then Table A is parent to Table B or Table A has child relationship with Table B
31.1.2. CAP Theorem - Consistency, Availability & Partition Tolerance
Relational Database (RDBMS) is CA MongoDB and document-oriented db is CP
31.1.3. Vertical and Horizontal Scaling
- Vertical scaling
- increase capacity of a single server by adding RAM, CPU or storage space
- Horizontal scaling
- divide the dataset and load over multiple additional servers
31.1.4. Modeling Optimization
- First normal form (1NF)
- every field should have one value. (Not comma separated values)
- Second normal form (2NF)
- If you have multiple primary keys (composite keys) in a table, every field should be unique to all of those composite keys
- e.g. CourseID and Date are 2 primary keys in table A. Field
CourseTitleis unique to courseID. courseTitle should not be in table A. Create a new table B to include CourseID and CourseTitle
- e.g. CourseID and Date are 2 primary keys in table A. Field
- Third normal form (3NF)
- Say you have CourseID, Date, Room, Seats in table A. Seats is unique to Room. You should create table B to hold Room, Seats and include just Room in table A
31.1.5. Integrity mode - ACID and BASE
- ACID
- usually what relational DB provides
- Atomic
- All parts involved should be either success or failure as a whole
- Consistent
- cannot result in a situation that violates any of the integrity rules
- Isolated
- all relevant data is locked until the transaction is finished. (Race condition)
- Pessimistic Locking
- Once locked, others can't even read (be refused or have to wait)
- Optimistic Locking
- Once one request lock data, other requests can read but roll back when try to write to db while lock is encountered
- Durable
- Once success, always success
- BASE
- usually NoSQL provides
- Basic availability
- Soft-state
- Eventual consistency
31.1.6. Index
One table can have only one clustered index which is by default the primary key column. You may make another column as non-clustered index. Each non-clustered index creates a new table which includes the clustered index column and the new column (sorted). When a new row is added to the main table, a new row is also added to each non-cluster indexed table.
31.2. Access
31.2.1. Null
IsNull(col1) return true or false Nz(col1,'') return '' if col1 is null
Sometimes IsNull() and Nz() don't work, e.g. UcanAccess, Use NVL() which is the same as Nz(). Or use col1 IS NULL
31.2.2. Coldfusion Datetime
<cfset dStart = DateFormat(form.startdate, "mm/dd/yyyy")> SELECT * FROM aTable WHERE DateStart >= #CreateDate(Year(now()),Month(now()),Day(now()))# AND DateEnd <= #CreateDate(Year(now()),Month(now()),DaysInMonth(now()))# AND DateStart BETWEEN #dStart# AND #CreateDate(Year(dStart),Month(dStyart),DaysInMonth(dStart))#
31.2.3. De-dup
DELETE t1.* FROM abc t1 WHERE t1.id NOT IN ( SELECT TOP 1 id FROM abc t2 WHERE t1.col1 = t2.col1 AND t1.col2 = t2.col2 ORDER BY t2.datefield DESC, t2.id )
31.2.4. Random Order
SELECT * FROM aTable ORDER BY rnd(INT(NOW*id)-NOW*id) <!--- OR rnd(aTable.ID) --->
31.2.5. Troubleshooting
31.2.5.1. "Record is too large"
This is because the size of a single record is around 2K. You can change the data type for some of fields to Long Text (>Access 2013) or Memo (<= Access 2000)
31.2.5.2. "Application uses a value of the wrong type for the current operation"
It's possibly due to you are using Microsoft Access with Unicode driver in ColdFusion and you insert to a memo or long text field.
cf_sql_longvarchar doesn't work. You should use <cfqueryparam cfsqltype="CF_SQL_CLOB" value="#description#"> instead.
31.3. MySQL
31.3.1. Version, Settings, Global variables
31.3.1.1. Get settings, global variables, session variable
-- all settings / global variables SHOW VARIABLES; -- one setting SHOW VARIABLES LIKE "%version%"; SHOW VARIALBES LIKE "%port%"; -- default character set for database -- MariaDB uses utf8mb4 as default charset SHOW VARIABLES LIKE 'char%'; -- default collation `collation_database` -- MriaDB default is utf8mb4_general_ci SHOW VARIABLES LIKE 'collation%'; -- Or just use one line get 2 variables SELECT @@character_set_database, @@collation_database; -- @@var_name means to get the session value if exists and the global value otherwise -- To get or set session value specifically SELECT @@SESSION.character_set_database; -- To get or set global value specifically SELECT @@global.character_set_database; -- SET default is to set session variable SET my_session_system_var_name = 'a'; -- eq. to SET @@my_session_system_var_name = 'a'; SET @@SESSION.my_session_system_var_name = 'a'; SET SESSION my_session_system_var_name = 'a'; -- SET global system variable SET @@GLOBAL.my_session_system_var_name = 'a'; SET GLOBAL my_session_system_var_name = 'a'; -- @ is User-defined variable -- default collation for a charset -- utf8mb4_general_ci -- refer to mysql:charset SHOW CHARACTER SET WHERE CHARSET = 'utf8mb4'; -- check if it's MariaDB SELECT VERSION(); -- Database table mysql SELECT User, Host, Password FROM mysql.user;
31.3.1.2. Config: mysqld
- 31.4.2
- default which means it only allows localhost connection
- allow all network access
- MySQL 8.0 may not be present. Add it
- Make sure to comment out
skip-networkingdirectives
- Make sure to comment out
- default 1 since 5.6.6
31.3.1.3. MySQL Mode (sql_mode)
SELECT @@SESSION.sql_mode;- Values
STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION- default
STRICT_TRANS_TABLES- Enable strict mode for transactional storage engines
- insert query should provide all values that are
not null - It includes
ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE
NO_ENGINE_SUBSTITUTION- disabled
CREATE TABLEuses default engine if not specified, andALTER TABLEthrows warning and the table is not altered- enabled
- an error occurs and the table is not created or altered if the desired engine is unavailable
- (no term)
ANSI_QUOTES- enabled
- double quotes cannot be used to quote literal strings. Both
"and`are identifier quote character
31.3.1.4. Allow remote access to user root
- mysql:plugin:auth_socket
Set priviledges
-- change 'password' to the password you want for the user root -- '%' means all host, this will add a row in table `user` of database `mysql` GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'password'; FLUSH PRIVILEGES; -- Then restart :: sudo systemctl restart mysql.service
May encounter `Your password does not satisfy the current policy requirements`
- The best way is to uninstall the validate_password plugin as changing the config file to change the policy level to low is not successful
mysql -uroot -pand then
uninstall plugin validate_password(I tried this and it worked) orUNINSTALL COMPONENT 'file://component_validate_password';- Restart MySQL
sudo systemctl restart mysql.service- Reference
- https://stackoverflow.com/questions/36301100/how-do-i-turn-off-the-mysql-password-validation
- (no term)
- The plugin was installed during
mysql_secure_installation - After uninstall, you shouldn't see any validate_password settings
SHOW VARIABLES LIKE 'validate_password%';
- Set 2 to allow all network access
31.3.1.5. Minimum Config
http://www.tocker.ca/2014/03/10/configuring-mysql-to-use-minimal-memory.html https://github.com/levonlee/vagrant/tree/master/build/docker-compose/mysql
[mysqld] innodb_buffer_pool_size=5M innodb_log_buffer_size=256K query_cache_size=0 max_connections=10 key_buffer_size=8 thread_cache_size=0 host_cache_size=0 innodb_ft_cache_size=1600000 innodb_ft_total_cache_size=32000000 # per thread or per operation settings thread_stack=131072 sort_buffer_size=32K read_buffer_size=8200 read_rnd_buffer_size=8200 max_heap_table_size=16K tmp_table_size=1K bulk_insert_buffer_size=0 join_buffer_size=128 net_buffer_length=1K innodb_sort_buffer_size=64K #settings that relate to the binary log (if enabled) binlog_cache_size=4K binlog_stmt_cache_size=4K
31.3.1.6. charset, collation mysql:collation
- Collation
- A set of rules that defines how to compare and sort character strings
- Every collation belongs to a single character set
- A character set has at least one collation and most have 2 or more collations
- case-insensitive at least in stopwords matching in full-text search
*_cs- case-sensitive
- Character Set
latin1- default for MySQL, 1 byte
utf8- default collation
utf8_general_ci, 3 bytes utf8mb4- MariaDB default, default collation
utf8mb4_general_ci, 4 bytes. PHP's UTF8 is 4 bytes
- Get all available character sets and their default collation
SHOW CHARACTER SET;Maxlencolumn shows each character byte
Get and set charset and collation
SELECT @@character_set_database, @@collation_database; ALTER DATABASE db_name CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE {tbl_name} CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE {tbl_name} MODIFY "field_name" "field_type" CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
- See MySQL Index
31.3.1.7. Memory, InnoDB Buffer, packet
- name
- default
- innodb-buffer-pool-size
innodb-buffer-pool-chunk-size * innodb-buffer-pool-instaces- max-allowed-packet
- 16M, total BLOB data size for a single row
31.3.1.8. Last update time for a table in a database
SELECT UPDATE_TIME, TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA NOT LIKE '%information_schema%' AND TABLE_SCHEMA LIKE '%your_db_name%' AND TABLE_NAME LIKE '%your_db_table_name%'
SELECT MAX(UPDATE_TIME), TABLE_SCHEMA FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA NOT LIKE '%information_schema%' GROUP BY TABLE_SCHEMA ORDER BY MAX(UPDATE_TIME)
31.3.1.9. Query Log
MySQL >= 5.1.12
show variables like 'general_log%'; // see if it's enabled show variables like 'log_output%'; // TABLE :: log to a table, FILE or NONE // Change this at runtime SET GLOBAL log_output = 'TABLE'; SET GLOBAL general_log = 'ON'; SELECT * FROM general_log // If you prefer to output to a file instead of a table: SET GLOBAL log_output = "FILE"; the default. SET GLOBAL general_log_file = "/path/to/your/logfile.log"; SET GLOBAL general_log = 'ON';
31.3.1.10. Threads
- Before v8.0.14, 1 query uses only 1 CPU core (concurrency not parallism)
- v8.0.14 uses multi CPU cores only in
SELECT COUNT(*) FROM t1without anyWHEREnorGROUP BY
- v8.0.14 uses multi CPU cores only in
- Concurrency threads can be configured
See settings
SELECT @@innodb_thread_concurrency, @@innodb_read_io_threads, @@innodb_write_io_threads;- default 0 means no limit. Any non-zero throttles
- default 4
- default 4
31.3.1.11. sql_mode
-- Remove a mode from sql_mode which is comma delimited SET SESSION sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));
31.3.2. mysql Client
31.3.2.1. Commands
- https://dev.mysql.com/doc/refman/8.0/en/mysql-commands.html
status- Current user, database, connection port, uptime, character set, MySQL Server version, command line delimiter
- display the result using vertical format
- open and edit query/command buffer
- After exit the buffer, type the delimiter
;and enter to run the command in the buffer
- After exit the buffer, type the delimiter
- print the current query/command buffer
31.3.2.2. mysql Client Options
- https://dev.mysql.com/doc/refman/8.0/en/mysql-command-options.html
Execute
mysql -e "SELECT * FROM a WHERE col1 LIKE '%some%'" -uroot -p mydbname > /path/to/bk.tsv- Escape
"inside the query \"- Escape
\ \\- (no term)
Use Heredoc
mysql -uroot -pABC mydb <<EOFSQL -- Avoid using backticks, escape with \` EOFSQL
- Escape
- don't print the row of columns
- batch mode. Print results using tab as column separator and new line as row separator (nontabular output). Default is column separator is
|with spaces padded and new line as row separator (tabular output) - For nontabular output, special characters are escaped. Disable this character escaping
- Newline
\n- Tab
\tNUL\0backslash\\
31.3.3. SSH Tunneling
# forward local port 9001 to mysshhost port 3306, login to the mysshhost shell ssh -L 9001:localhost:3306 mysshhost # In anothe shell mysql -uroot -p -P 9001 --protocol=tcp mydb # or use -h, but it cannot be localhost! mysql -uroot -p -P 9001 -h 127.0.0.1 mydb # Don't need SSH Tunneling for dump, the output is redirected to local backup.sql ssh mysshhost mysqldump -uroot -p [..options] mydb > backup.sql
31.3.4. Install
# Ubuntu apt-get update; apt-get install mysql-server service mysql start # or sudo systemctl restart mysql.service # Debian yum install php-mysql service httpd restart service mysqld start # both mysql_secure_installation # status systemctl status mysql.service
31.3.4.1. Manage users
- Refer to mysql:user:remote access
- auth_socket usually works in Linux where auth is done through Linux localhost users auth
- In Ubuntu, user root by default uses
auth_socketinstead ofmysql_native_password. Later is required for external db connection using user rootALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'my-strong-password';FLUSH PRIVILEGES;
- In Ubuntu, user root by default uses
-- see a list of all users SELECT user, host, password, authentication_string, plugin FROM mysql.user; CREATE USER 'sammy'@'localhost' IDENTIFIED BY 'password'; -- create a user that can connect to this server from any external IP address CREATE USER 'sammy'@'%' IDENTIFIED BY 'password'; -- CREATE USER 'sammy'@'192.168.100.%%' IDENTIFIED BY 'password'; -- grant all privileges to one database GRANT ALL PRIVILEGES ON mydatabase.* TO 'sammy'@'localhost'; -- same as above but the new user can also grant privileges to others GRANT ALL PRIVILEGES ON mydatabase.* TO 'sammy'@'localhost' WITH GRANT OPTION; -- grant all privileges to all tables/databases GRANT ALL PRIVILEGES ON *.* TO 'sammy'@'localhost' WITH GRANT OPTION; -- except for granting privileges for a brand new user, after INSERT, UPDATE, DELETE, flush is required FLUSH PRIVILEGES; -- show granted privileges SHOW GRANTS FOR 'sammy'@'localhost'; SHOW GRANTS; -- eq. to -- SHOW GRANTS FOR CURRENT_USER(); -- SHOW GRANTS FOR CURRENT_USER; -- Show grants for all users select * from information_schema.user_privileges; -- remove a user DELETE FROM mysql.user WHERE user = 'sammy';
31.3.4.2. revoke
REVOKE INSERT ON *.* FROM 'li'@'localhost'; REVOKE ALL, GRANT OPTION FROM 'li'@'localhost'; -- it does not remove the user from mysql.user table -- DROP USER :: remove one or more user accounts and all grants of them SHOW GRANTS;
31.3.4.3. Create database
CREATE DATABASE `dbname_in_lowercase`;
USE `dbname_in_lowercase`;
SHOW TABLES;
31.3.4.4. MySQL does not start after installation, Ubuntu
No directory, logging in with HOME=/ and service mysql status shows it is inactive.
This is new.. Change the owner:group for relevant folders
chown -R mysql:mysql /var/lib/mysql /var/run/mysqld
31.3.4.5. Reset root password, Ubuntu
ERROR 1045: Access denied for user: 'root@localhost' (Using password: NO) or yes
# stop sudo /etc/init.d/mysql stop # skip permission check sudo /usr/sbin/mysqld --skip-grant-tables --skip-networking & # start mysql client without a password mysql -u root FLUSH PRIVILEGES; # Update root password SET PASSWORD FOR root@'localhost' = PASSWORD('password'); # If you have a mysql root account that can connect from everywhere, you should also do: UPDATE mysql.user SET Password=PASSWORD('password') WHERE User='root'; # Alternate USE mysql UPDATE user SET Password = PASSWORD() WHERE Host = 'localhost' And User='root'; # Alternate for root can access from everywhere USE mysql UPDATE user SET Password = PASSWORD('password'); WHERE Host = '%' AND User = 'root'; FLUSH PRIVILEGES; exit service mysql restart # or sudo /etc/init.d/mysql stop sudo /etc/init.d/mysql start
31.3.5. MySQL 8.0
31.3.6. Data Types
31.3.6.1. Integer
NOT NULLdefault- 0
- SIGNED
- contains negative, 0 and positive
- UNSIGNED
- contains 0 and positive
- TINYINT
-
- 1 byte
- -128 to 127
- 0 to 255
- m is just display-column width. Storage size is the same
- PHP always returns the same range as TINYINT
- (no term)
- SMALLINT
- 2 bytes
- -32,768 to 32,767
- 0 to 65,535
- (no term)
- MEDIUMINT
- 3 bytes
- -8,388,608 to 8,388,607
- 0 to 16,777,215
- (no term)
- INT
- 4 bytes
- -2,147,483,648 to 2,147,483,647
- 0 to 4,294,967,295
- The max positive number to Unix Timestamp is Sun Feb 07 2106 06:28:15 GMT+0000
- m is just display-column width. Storage size is the same
- (no term)
- BIGINT
- 8 bytes
- -2^63 to 2^63-1
- 9.223x10^19
- 9.223MMM
- 0 to 2^64-1
- 18.447x10^19
- 18.447MMM
31.3.6.2. Decimal
DECIMAL(M,D)- In MySQL,
NUMERIC(M,D)is exactly the same asDECIMAL(M,D) - default 10. Max 65. total number of digits
- default 0. Max 30. number of digits to the right of the decimal point. It should not be larger than M
- In MySQL,
- Range of -999 to 999, 3 digits, total 2 bytes
- 9 digits on either side of the decimal point (the scale). Total (2*9)/9*4 = 8 bytes
- 14 digits (9+5) on the left, 6 digits on the right. Total (4+3) + (3) = 10 bytes
- 18 digits (9+9) on the left, 2 digits on the right. Total (4+4) + 1 = 9 bytes
- Up to trillion dollars. 9+4 on left and 2 digits on right. 4+2 + 1 = 7 bytes
- Every 9 digits is 4 bytes, the remaining digits left over require:
- 1 to 2 digits left over
- 1 byte
- 3 to 4
- 2 bytes
- 5 to 6
- 3 bytes
- 7-9
- 4 bytes
Insert or Update a decimal field with empty string
-- Throws a warning but the row is inserted with value 0.00 INSERT INTO myTable (decimal_field) VALUE ('');
UNSIGNEDfor float, double and decimal are deprecated MySQL 8+.After upgraded to MySQL 8+
-- remove unsigned alter table my_table modify my_col decimal(20, 2) default null; alter table my_table add constraint my_col_unsigned check (my_col > 0);
31.3.6.3. BOOL and BOOLEAN
- Both are synonyms for TINYINT(1). Refer to mysql:tinyint
31.3.6.4. Datetime mysql:datatype:datetime
Get global and sessiond timezones
SELECT @@GLOBAL.time_zone, @@SESSION.time_zone; -- List of all timezones SELECT * FROM mysql.time_zone_name;
If it's
SYSTEM, refer to linux:timezone- format 'HH:MM:SS' from '-838:59:59.000' to '838:59:59.000'
- return string e.g.
2019-07-12 01:02:03in session timezone - returns a string
YYYY-MM-DD hh:mm:ssor numericYYYYMMDDhhmmssdepending on the context- fsp argument
- a fractional seconds precision from 0 to 6
- (no term)
- In session timezone
- Date comparison
- 1 Year ago
post_date > DATE_SUB(NOW(), INTERVAL 1 YEAR)
- return integer or float or NULL. Convert the string of time in session timezone to UTC integer
- date
- maybe a
- DATE
- DATETIME
- TIMESTAMP string, see
NOW()UNIX_TIMESTAMP('2024-01-19T23:03:48.676Z')1705705428.676
- a number in YYMMDD, YYMMDDhhmmss, YYYYMMDD, or YYYYMMDDhhmmss format
-- Custom datetime TIMESTAMP('2003-12-31') CAST(post_date AS DATE) > '2014-01-01' FROM_UNIXTIME(intTime) > '2014-01-01' FROM_UNIXTIME(intTime) BETWEEN '2014-01-01 00:00:00' AND '2014-12-31 23:59:00' SELECT NOW(); -- datetime type -- UNIX Time -- default date format DATE_FORMAT(FROM_UNIXTIME(intTime),'%Y-%m-%d %H:%i:%s') UNIX_TIMESTAMP(NOW()) -- returns integer, accepts date type -- both using timezone name, requires MySQL to support the timezone name SELECT UNIX_TIMESTAMP(CONVERT_TZ('2022-04-01 11:50:00', 'America/Toronto', 'UTC')); -- one using timezone name, requires MySQL to support the timezone name SELECT UNIX_TIMESTAMP(CONVERT_TZ('2022-04-01 11:50:00', '-04:00', 'UTC')); -- Guaranteed to work -- The input date is of time zone '-04:00'. Convert timezone from '-04:00' to session timezone SELECT CONVERT_TZ('2022-04-01 11:50:00', '-04:00', @@SESSION.time_zone); -- eq. SELECT CONVERT_TZ('2022-04-01 11:50:00', '-04:00', '+00:00'); -- eq. if session timezone is equal to system (e.g. Linux that MySQL is installed on) timezone SELECT CONVERT_TZ('2022-04-01 11:50:00', '-04:00', 'SYSTEM'); SELECT * FROM a WHERE aUnixTimeColumn > UNIX_TIMESTAMP(NOW() - INTERVAL 24 HOUR) -- last 24 hours -- From UNIX timestamp (int) to a specific timezone, return date type SELECT CONVERT_TZ(FROM_UNIXTIME(aColumnOfUnixTimeStamp), 'UTC', '-04:00');
31.3.6.5. String Functions
- https://dev.mysql.com/doc/refman/5.7/en/string-functions.html
- visual string length
byte length/size
-- Times a substring appear in a string. e.g. substring is lili SELECT ROUND((LENGTH(col1) - LENGTH(REPLACE(col1, 'lili', ''))) / LENGTH('lili')) AS numOfAppearanceOfSubstring FROM t1
- Regex search
-- column is integer SELECT field FROM myTable WHERE field REGEXP '^-?[0-9]+$';
REGEXP_REPLACE
Do more complicated find and replace. MariaDB driver on host is required
SELECT REGEXP_REPLACE("stackoverflow", "(stack)(over)(flow)", '\\2 - \\1 - \\3');
- String delimited list
substring_index ( substring_index ( context,',',1 ), ',', -1) ) substring_index ( substring_index ( context,',',2 ), ',', -1) ) substring_index ( substring_index ( context,',',3 ), ',', -1) ) substring_index ( substring_index ( context,',',4 ), ',', -1) ) -- it means 1st value, 2nd, 3rd, etc.
- Explanation
- original string is "34,7,23,89",
substring_index( context,',', 3)returns "34,7,23". The outer substring_index takes the value returned by the inner substring_index and the -1 allows you to take the last value. So you get "23" from the "34,7,23" - (no term)
- See mysql:string:list:temp table for more
FIELD(str, str1, str2, ...)
- return index position (starting from 1) of
strin thestr1, str2, ...list- 0 if not found or
strisNULL
- 0 if not found or
- If all arguments including
strare strings, then compare as strings. If all arguments are numbers, compare as numbers- Otherwise, compare as double
- A good place to use it is in
ORDER BY - The complement is
ELT()ELT(N, str1, str2, ... )SELECT ELT(1, 'Aa', 'Bb', 'Cc', 'Dd')'Aa'
- return index position (starting from 1) of
CONCAT(str1, str2, ...)
If there is any value that is
NULL, CONCAT returnsNULLSET @s3 = 'http://s3.amazonaws.com/abc/'; SELECT CONCAT('t1', null); -- null SELECT REGEXP_REPLACE(thumbnail.uri, "^(s3://)(.*)", CONCAT(@s3,'\\2')) AS thumbnail';
Multiple columns all have to have values that are not empty string nor null
SELECT * FROM aTable WHERE 1 = 1 AND CONCAT(IF(TRIM(colA), 1, NULL), IF(TRIM(colB), 1, NULL), IF(TRIM(colC), 1, NULL));
- See 31.3.18.1
- See FIND_IN_SET()
CONCAT_WS(separator, str1, str2, ...)
Concatenate columns or strings of one row. It doesn't skip empty string but it does skip NULL values
CONCAT_WS(',', colA, colB, 'Column C Value') WHERE url_alias.source = CONCAT('node/', n.nid) CONCAT_WS(',', NULL, '123', NULL) -- 123 CONCAT_WS(',', NULL, NULL) -- empty string CONCAT_WS(',', NULL, '123', '', NULL) -- 123,, CONCAT_WS(',', NULL, '123', TRIM(NULLIF(' ', '')), NULL) -- 123
- Replace all occurances, case-sensitve. multibyte safe
SELECT REPLACE('www.mysql.com', 'w', 'Ww');
LIKE
expr LIKE pattern [ESCAPE 'escape_char']Case insensitive. String comparisons are not case-sensitive unless one of the operands is case-sensitive (uses a case-sensitive collation or is a binary string)
SELECT 'abc' LIKE 'ABC'; -- -> 1 SELECT 'abc' LIKE _utf8mb4 'ABC' COLLATE utf8mb4_0900_as_cs; -- -> 0 SELECT 'abc' LIKE _utf8mb4 'ABC' COLLATE utf8mb4_bin; -- -> 0 SELECT 'abc' LIKE BINARY 'ABC'; -- -> 0
- default
\ expr NOT LIKE pat [ESCAPE 'escape_char']eq.NOT (expr LIKE pat [ESCAPE 'escape_char'])
- eq.
*- Ends with
SELECT * FROM aTable WHERE colA LIKE '*.pdf'
- like
? - use
\\n- newline character.
\\nto escape\n \- Use
\\\\to escape, if the target\is at the end of the pattern string, use\\
SOUNDDEX(),SOUNDS LIKESUBSTRING()
- alias SUBSTR
SUBSTR(str,pos) SUBSTR(str FROM pos) SUBSTR(str,pos,len) SUBSTR(str FROM pos FOR len) -- The first 10 characters of a column LEFT(aColumn, 10) = SUBSTRING(aColumn, 1, 10) -- The last 10 characters of a column RIGHT(aColumn, 10) = SUBSTRING(aColumn, -10) SUBSTRING('foobarbar' FROM 4) = SUBSTRING('foobarbar', 4) = 'barbar' -- Get 6 characters from the 5 + 1 position SUBSTRING('Quadratically',5,6) = SUBSTRING('Quadratically' FROM 5 FOR 6) = 'ratica'
- LOCATE()
- multibyte safe and is case-sensitive only if at least one argument is a binary string
SELECT LOCATE('bar', 'foobarbar') = 4; SELECT LOCATE('xbar', 'foobar') = 0; -- search 'bar' from 5 + 1 position SELECT LOCATE('bar', 'foobarbar', 5) = 7;
- MATCH … AGAINST
- https://dev.mysql.com/doc/refman/8.0/en/fulltext-search.html
- Can be used in
SELECT,GROUP BY,HAVINGorORDER BY exprinAGAINSTis a search string which must be a string value that is constant during query evaluation. This rules out, for example, a table column because that can differ for each row- On top of stopwords, words that have length less than
SELECT @@innodb_ft_min_token_size = 3;and larger thanSELECT @@innodb_ft_max_token_size = 84;are not in the fulltext index- Do not apply to FULLTEXT indexes created using the ngram parser. ngram token size is defined by the
ngram_token_sizeoption - Do not apply to the search string with leading plus and minus sign
'+word +the'
- Do not apply to FULLTEXT indexes created using the ngram parser. ngram token size is defined by the
- Returns the matching score, the higher the more relevant the matching is
MATCH (col1,col2,...) AGAINST (expr [search_modifier]) search_modifier: { IN NATURAL LANGUAGE MODE | IN NATURAL LANGUAGE MODE WITH QUERY EXPANSION | IN BOOLEAN MODE | WITH QUERY EXPANSION }
- FULLTEXT index type
- Only for
CHAR,VARCHARorTEXTcolumns
CREATE FULLTEXT INDEX myfulltextindex ON mytable (field1, field2, field3);
- Only for
- Stopwords
- Stopwords are not part of full text index and are ignored in case of a linguistic search
- Default, it means to use stopwords from this order
SELECT @@innodb_ft_user_stopword_tableSELECT @@innodb_ft_server_stopword_table- Default set of stopwords
- Boolean Mode
- https://dev.mysql.com/doc/refman/8.0/en/fulltext-boolean.html
- InnoDB
Support
- leading plus or minus signs
- whether two or more words all start within a specified distance from each other, measured in words
MATCH(col1) AGAINST('"word1 word2 word3" @8' IN BOOLEAN MODE)
- Does not support
'+*''+-''+-apple'
AGAINST ('+WordMustHave -WordMustNotHave WordNiceToHave' IN BOOLEAN MODE);
FIND_IN_SET
strlistis a comma separated string- Return
- 0
- if
strnot instrlist - null
- if either
strorstrlistis NULL
FIND_IN_SET(str, strlist) -- 2 SELECT FIND_IN_SET('b', 'a,b,c,d'); -- 0 SELECT FIND_IN_SET('b', 'a,b ,c,d'); SELECT FIND_IN_SET('b', 'a, b,c,d'); -- Change delimiter SELECT FIND_IN_SET('b', REPLACE('a|b|c|d', '|', ',')); -- Add to list (requires column users.groups is non-null) UPDATE users SET groups = CONCAT(groups, '|account') WHERE userid = 1234 AND FIND_IN_SET('account', REPLACE(groups, '|', ',')) = 0; -- Remove from list (remove 2 from comma-delimited list, and list can contain multiple 2's) -- - Requires columnn mytable.categories is non-null UPDATE mytable SET categories = TRIM(BOTH ',' FROM REPLACE( REPLACE( CONCAT( ',', REPLACE(categories, ',', ',,'), ',' ), ',2,', '' ), ',,', ',' ) ) WHERE FIND_IN_SET('2', categories)
31.3.6.6. CHAR and VARCHAR
- 1 character 1 byte
CHAR(n)- Fixed byte
- Required no storage space. So
CHAR(n)is n bytes - When retrieved, trailing spaces are removed unless
PAD_CHAR_TO_FULL_LENGTHmode is enabled- Trailing spaces are not retained when fetched!
- right padding with space
VARCHAR(n)- Variable length
- n is called column length and can be from 0 to 65,535
- Not padded thus trailing spaces are retained
- VARCHAR(3) and insert a column length 4 value and with trailing spaces, the value (the trailing spaces) will be truncated and saved. If the value has no trailing spaces, it will throw an error
- 1-byte or 2-byte length prefix plus actual data
- Length prefix
- If actual data =< 255 bytes, 1 storage byte
- if actual data > 255, 2 storage byte
- see 31.3.1.6
- Length prefix
31.3.6.7. TEXT Type
TEXTdata is not stored in the database server’s memory. QueringTEXTdata requires reading from disk, much slower thanCHARandVARCHAR- It holds data that is nonbinary strings
- It doesn't count towards the maximum row size for each record
- No padding on insert and no bytes are stripped on select
- If it is indexed, index entry comparisons are space-padded at the end
- Insert with exceeding max length will be truncated
- Cannot have DEFAULT value
- 255 characters, 255 Byte
- 65,535 characters, 64kb
- 16,777,215, 16mb
- 4,294,967,295 characters, 4GB
31.3.6.8. Blob
- Binary large object for variable amount of data
- Holds binary strings or so called byte strings
- Data is stored on hard disk rather than in memory
- TINYBLOB
- BLOB
- MEDIUMBLOB
- LONGBLOB
31.3.6.9. Null mysql:datatype:null
IFNULL(column_name, 0) WHERE column_name IS NOT NULL
31.3.6.10. UUID and GUID mysql:uuid
- https://mysqlserverteam.com/storing-uuid-values-in-mysql-tables/ https://mysqlserverteam.com/mysql-8-0-uuid-support/
- GUID is UUID. GUID is one implementation (Microsoft's) of UUID
- 36 numeric digits in 5 groups separated by
-8+4+4+4+12+4(hypens)=36. e.g.12345678-1234-1234-1234-123456789012 - In MySQL, it's
binary(16), In MSSQL (T-SQL), it'suniqueidentifier
MySQL 8.0 adds a few functions
CREATE TABLE t (id binary(16) PRIMARY KEY); INSERT INTO t VALUES(UUID_TO_BIN(UUID())); SELECT BIN_TO_UUID(id) FROM T; -- the records are inserted in random locations in the index tree which impacts performance -- To insert records in order, better to do it like this INSERT INTO t VALUES(UUID_TO_BIN(UUID(), true));
MySQL 5.6
- Use mariadb:uuid instead of the following
- Treat the following as the last resort
- The following is not performance optimized
CREATE TABLE users(id_bin binary(16), name varchar(200)); INSERT INTO users VALUES (UNHEX(REPLACE(UUID(),'-','')), 'Andromeda'); -- id_text is a virtual column and doesn't occupy disk CREATE TABLE users( id_bin binary(16), id_text varchar(36) GENERATED always as (insert( insert( insert( insert(hex(id_bin),9,0,'-'), 14,0,'-'), 19,0,'-'), 24,0,'-') ) virtual, name varchar(200)); INSERT INTO users (id_bin,name) VALUES(UNHEX(REPLACE(UUID(),'-','')), 'Andromeda'); -- If id_text is not an index, don't use WHERE ~WHERE id_text=text_form_of_XYZ~ -- If id_text is made to an index column, then it occupies space ALTER TABLE users ADD UNIQUE(id_text);
31.3.7. Index, Composite Index
- Index columns used in
WHERE,ORDER BYandGROUP BYclauses - A column with <= 767 bytes can be indexed in Antelope file format which is default for MySQL 5.6 or lower
varchar(255)inutf8charset max byte size 255*3=765varchar(191)inutf8mb4charset max byte size 191*4=764Barracuda file format and DYNAMIC row format become the default since MySQL 5.7.7, which allows index key prefixes up to 3072 bytes (Barracuda + COMPRESSED row format can also reach)
- becomes available on mysqld (MySQL) server version > 5.5.14 and default since 5.7.7
Use these settings in my.cnf and restart MySQL
[mysqld] # innodb_large_prefix is to be deprecated. True to use larger index prefix length than 767 bytes innodb_large_prefix=true innodb_file_format=barracuda innodb_file_per_table=true
- As of Nov 25, 2016, Pantheon MySQL (innodb) version is 5.6.26 and does not have the above setting
- If changing MySQL server setting is not possible and the index column has to be
utf8mb4, then need to convert those columns in every table to havevarchar(191). Which is what WordPress does
ALTER TABLE tbl_name CHANGE index_col index_col VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
- Referred by d7:mysql:charset
- See Index Hint
- Some MySQL 8 tricks to troubleshoot
31.3.7.1. Composite index
- An index on multiple columns. MySQL supports up to 16 columns
- also called multiple column index
Table definition
CREATE TABLE table_name ( c1 data_type PRIMARY KEY, c2 data_type, c3 data_type, c4 data_type, INDEX index_name (c2,c3,c4) );
Or create index directly
CREATE INDEX index_name ON table_name(c2,c3,c4);
- Use leftmost prefix to identify whether a combo is optimized
- Applies also for
GROUP BYANDDISTINCT- Only these aggregated functions are allowed
MIN()andMAX()with or withoutGROUP BY- With no
GROUP BY- AVG(DISTINCT)
- one single argument
- SUM(DISTINCT)
- one single argument
- COUNT(DISTINCT)
- multiple column arguments
- Constants must be used and operator is
=- Range like
>causes the index to be partially used which is called Tight Index Scan.- Both Loose Index Scan (better) and Tight Index Scan (good) prevent creating temporary tables
Filling gaps in the leftmost prefix is another way to achieve Tight Index Scan
-- There is a gap in the GROUP BY, but it is covered by the condition c2 = 'a': SELECT c1, c2, c3 FROM t1 WHERE c2 = 'a' GROUP BY c1, c3; -- The GROUP BY does not begin with the first part of the key, but there is a condition that provides a constant for that part: SELECT c1, c2, c3 FROM t1 WHERE c1 = 'a' GROUP BY c2, c3;
- Range like
- If Loose Index Scan is applicable,
EXPLAINwill showUsing index for group-byin theExtracolumn- Or in JSON
using_index_for_group_by
- Only these aggregated functions are allowed
- Index on 3 columns c2, c3, c4
- optimize queries using these combinations given that operator
=is only used- c2
- c2, c3
WHERE c2 = 1 AND c3 = 3WHERE c2 = 1 AND (c3 = 3 OR c3 = 4)
- e.g.
WHERE c2 = 1 AND c3 = 3 AND c4 = 4
- Notice, not optimized for the following
WHERE c3 = 3
- Index is not fully used
WHERE c2 = 2, c4 = 4- The index is used up to
c2
- The index is used up to
WHERE c2 = 2 AND c3 > 3 AND c4 = 4- operator is
>(range), so the onlyc2is used for the index
- operator is
- c2 is the most selective column meaning given c2, we can get more unique records
- Redundant indexes. We have an existing composite index
c2, c3, c4. The following are redundant indexesc2c2, c3
- optimize queries using these combinations given that operator
- Applies also for
31.3.7.2. Index Hint
- Index hints can be used in
SELECTqueries to specify whether or not to use the index column- https://dev.mysql.com/doc/refman/8.0/en/index-hints.html
- With no index hint, MySQL will smartly decided whether to force using index
- In fact, the less the number of query matched records are, the faster the speed is. Because for each selected record, MySQL has to bind the indexed column to the result
- Each hint requires index names, not column names
SHOW INDEX FROM myTable- This also shows the cardinality of each index
- Estimated number of uniquevalues in the index.
- The higher, the greater the chance that the query optimizer uses the index for lookups
- This also shows the cardinality of each index
- Index hints apply only to
SELECTandUPDATEstatements - The
FORCE INDEXhint acts likeUSE INDEX (index_list), with the addition that a table scan is assumed to be very expensive. In other words, a table scan is used only if there is no way to use one of the named indexes to find rows in the table. Syntax
tbl_name [[AS] alias] [index_hint_list] index_hint_list: index_hint [index_hint] ... index_hint: USE {INDEX|KEY} [FOR {JOIN|ORDER BY|GROUP BY}] ([index_list]) | {IGNORE|FORCE} {INDEX|KEY} [FOR {JOIN|ORDER BY|GROUP BY}] (index_list) index_list: index_name [, index_name] ...
Samples
SELECT * FROM table1 USE INDEX (col1_index,col2_index) WHERE col1=1 AND col2=2 AND col3=3; SELECT * FROM table1 IGNORE INDEX (col3_index) WHERE col1=1 AND col2=2 AND col3=3; SELECT * FROM t1 USE INDEX (i1) IGNORE INDEX FOR ORDER BY (i2) ORDER BY a;
31.3.8. Common queries, functions, operators
31.3.8.1. Reuse SELECT alias
- https://dev.mysql.com/doc/refman/8.0/en/select.html
- SELECT aliases (or column aliases) can only be used in later columns in the same
SELECT,ORDER BY,GROUP BYandHAVING, notWHERE!- Order
FROMSELECTaliases cannot be used
WHERESELECTaliases cannot be used
SELECT select_expr1 AS myAlias1, select_expr2 AS myAlias2- Aliases can be used in later
GROUP BY,HAVINGandORDER BYwith some exceptions- User-defined variables
- The order of evaluation for expressions involving user variables is undefined. For example,
- there is no guarantee that
SELECT @a, @a:=@a+1evaluates @a first and then performs the assignment.
- there is no guarantee that
- The order of evaluation for expressions involving user variables is undefined. For example,
- User-defined variables
An earlier alias can be used in later
select_exprSELECT s.f1 + s.f2 AS total_sale, s.f1 / (SELECT total_sale) AS f1_percent IF((SELECT total_score) > 100, 1, 0) AS good_or_bad, IF((SELECT good_or_bad), 'good score', 'bad score') AS good_or_bad_text, -- Limitation of this method ~(SELECT my_alias)~ : my alias should not be a group/aggregation function result. e.g. SELECT COUNT(*) AS localCount, (SELECT localCount) + 1
- Aliases can be passed onto only the first subquery not to the next deeper level subqueries in later
select_expr- Use
JOIN
- Use
- Aliases can be used in later
GROUP BYHAVING- Aggregated values from the
SELECT - Keys from the
GROUP BY - Functions that take no argument e.g. current date
- Expressions composed of the above
- Besides only the above, MySQL has an extension to standard SQL:
- permits HAVING to refer to columns in the
SELECTlist and columns in outer subqueries as well.
- permits HAVING to refer to columns in the
- Aggregated values from the
SELECTORDER BYselect_expralias can be used, it's picked first beforeFROMtable.aCol
LIMIT- Constants
- Local variables
- Order
- In
HAVING- PhpStorm might report the following if an alias of aggregate function result is used
- Using aggregate-free condition(s) in HAVING clause might be inefficient. Consider moving them to WHERE
- PhpStorm might report the following if an alias of aggregate function result is used
- SELECT alias CANNOT be used in
WHEREandFROM, need subquery
- Use sub query
- create a sub table with additional computed fields added in original table, and select from it later
SELECT tbl1.total_sale, tbl1.f1 / tbl1.total_sale AS f1_percent FROM (SELECT s.f1 + s.f2 AS total_sale, s.f1 FROM sales s) AS tbl1;
31.3.8.2. Select all columns except some columns
SET @sql = CONCAT( 'SELECT ', (SELECT REPLACE(GROUP_CONCAT(COLUMN_NAME), '<columns_to_omit>,', '') FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '<table>' AND TABLE_SCHEMA = '<database>'), ' FROM <table>'); PREPARE stmt1 FROM @sql; EXECUTE stmt1;
31.3.8.3. IN
SELECT userid, created_by FROM users WHERE userid in ($x); -- eq. SELECT userid, created_by from users WHERE $x in (userid); SELECT userid FROM users WHERE login IN ($x) OR dispname OR ($x) -- eq. SELECT userid FROM users WHERE $x IN (login,dispname)
31.3.8.4. JOIN
- In MySQL,
JOIN,CROSS JOIN, andINNER JOINare the same.- In standard SQL, they are not equivalent.
INNER JOINis used with anONclause,CROSS JOINis used otherwise.CROSS JOINdoes not haveON- Table A of n rows
CROSS JOINTable B of m rows will result n*m rows
- In standard SQL, they are not equivalent.
- Comma separated tables is eq. to
JOINbut the precedence is less thanJOIN,INNER JOIN,LEFT JOIN,CROSS JOIN- Generally, avoid mixing comma separated tables with
* JOIN *
- Generally, avoid mixing comma separated tables with
USINGvsONa LEFT JOIN b USING (c1, c2, c3) a LEFT JOIN b ON a.c1 = b.c1 AND a.c2 = b.c2 AND a.c3 = b.c3 -- SELECT * -- USING is COALESCE(a.c1, b.c1), COALESCE(a.c2, b.c2), COALESCE(a.c3, b.c3) -- ON is a.c1, a.c2, a.c3, b.c1, b.c2, b.c3
JOINsubquery- The only place to pass any parent columns is in
ON. In other words, the subquery result has to be determined and then we can join this subquery
- The only place to pass any parent columns is in
- Join on one row only
- Tables
- users
- id
- widgets
- id
- one to many
- users
-- In general, using ORDER BY ... LIMIT 1 is slower compared to MAX() or MIN() SELECT * FROM users JOIN ( SELECT * FROM widgets WHERE id IN ( SELECT MAX(id) FROM widgets GROUP BY user_id ) ) AS most_recent_user_widget ON users.id = most_recent_user_widget.user_id -- Multiple columns sort SELECT * FROM users JOIN ( SELECT * FROM widgets WHERE CONCAT_WS(0x1F, col2, id) IN ( SELECT MAX(CONCAT_WS(0x1F, col2, id)) FROM widgets GROUP BY user_id ) ) AS most_recent_user_widget ON users.id = most_recent_user_widget.user_id -- Group by but order by non-grouped columns SELECT t1.*, t2.final_exam_score FROM ( -- Original query SELECT scid, SUM(score) FROM s_score GROUP BY scid ) t1 JOIN ( -- Join on 1 row SELECT * FROM s_score WHERE CONCAT_WS(0x1F, final_exam_score, ssid) IN ( SELECT MAX(CONCAT_WS(0x1F, final_exam_score, ssid)) FROM s_score GROUP BY scid ) ) t2 ON t1.scid = t2.scid; -- Faster SELECT t1.*, t2.final_exam_score FROM ( -- Original query SELECT scid, SUM(score), MAX(CONCAT_WS(0x1F, final_exam_score, ssid)) AS _sortColumns FROM s_score GROUP BY scid ) t1 JOIN ( -- Join on 1 row SELECT * FROM s_score ) t2 ON t1._sortColumns = CONCAT_WS(0x1F, t2.final_exam_score, t2.ssid);
- Tables
- Join on the 2nd, 3rd, 4th row
MySQL 8.0 can use ROW_NUMBER https://www.mysqltutorial.org/mysql-window-functions/mysql-row_number-function/
ROW_NUMBER() OVER (<partition_definition> <order_definition>) partition_definition: PARTITION BY <expression>,[{,<expression>}...] order_definition: ORDER BY <expression> [ASC|DESC],[{,<expression>}...] SELECT productLine, productName, quantityInStock, ROW_NUMBER() OVER ( PARTITION BY productLine ORDER BY quantityInStock DESC) row_num FROM products
In MySQL 5, to emulate the ROW_NUMBER
-- For each filtered log, find the 3rd, 2nd or 1st action date in the raw log conditionally depening on filtered log totalComms as the charge date SELECT T1.*, _FindChargeDate._chargeDate FROM filteredlog T1 LEFT JOIN ( SELECT _RawLog.actiondate AS _chargeDate, CASE WHEN _RawLog.action IN (1, 2, 5) THEN 1 -- Action Type 1 WHEN _RawLog.action IN (3, 4, 6) THEN 2 -- Action Type 2 ELSE 0 END AS _actionType, _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4, @rowNumberCalculated := CASE WHEN @rowNumberHashCalculated = CONCAT_WS('|', _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4, (SELECT _actionType)) THEN @rowNumberCalculated + 1 ELSE 1 END AS _rowNumber, @rowNumberHashCalculated := CONCAT_WS('|', _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4, (SELECT _actionType)) FROM rawLog AS _RawLog, (SELECT @rowNumberHashCalculated := '', @rowNumberCalculated := 0) AS _rowRank ORDER BY _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4, _actionType, _RawLog.actiondate ) AS _FindChargeDate ON _FindChargeDate.col1 = T1.col1 AND _FindChargeDate.col2 = T1.col2 AND _FindChargeDate.col3 = T1.col3 AND _FindChargeDate.col4 = T1.col4 AND _FindChargeDate._actionType = T1._actionType AND _FindChargeDate._rowNumber = IF(T1.totalComms > 2, 3, IF(T1.totalComms > 1, 2, 1))
31.3.8.5. GROUP BY and HAVING
- After
WHEREbeforeORDER BYandHAVINGWHERE 1=1 GROUP BY col1 HAVING fieldDefinedinSELECT > 1 ORDER BY col1
GROUP BY col1 ORDER BY NULLORDER BYon non-grouped columns, see 31.3.8.4.1, the grouped query result on the left of the join, the order by table(s) is on the right of the join
31.3.8.6. COUNT
COUNT(*)- count all rows of a table
COUNT(columnA or expression)- count non-null only
- (no term)
- Use
SUM(CASE WHEN 0 = 0 THEN 1 ELSE 0 END)orSUM(IF(aColumn = 1, 1, 0))~or ~SUM(IFNULL(aColumn, 0))to achieve count if
31.3.8.7. REGEXP mysql:regex
Unless the field is binary, REGEXP is case-insensitive.
WHERE aut_name REGEXP "^w" WHERE aut_name REGEXP "^I am \"Great!\""
31.3.8.8. DELETE
DELETE FROM t1 WHERE id = 123; -- Delete records from t1 table only even though FROM is a join DELETE t1 FROM t1 JOIN t2 ON t2.id = t1.id WHERE t1.id = 123 AND t2.type = 'abc'; -- all matching rows of t1 and t2 are deleted DELETE t1,t2 FROM t1 JOIN t2 ON t2.id = t1.id WHERE t1.id = 123 AND t2.type = 'abc';
31.3.8.9. INSERT
INSERT ... SELECT
If there's no record from
SELECT, no row is insertedINSERT INTO t1 (col1, col2) SELECT t2.id, t3.name FROM t2 JOIN t3 ON t2.t3bridge = t3.id WHERE t2.score > 100;
INSERT INTO timezones (id, timezonename) SELECT 25, 'America/Phoenix' WHERE NOT EXISTS ( SELECT * FROM timezones WHERE id = 25 )
INSERT INTO timezones (timezoneid, timezonename) (SELECT 25, 'America/Phoenix' FROM (SELECT 1 AS whocares) AS t WHERE NOT EXISTS (SELECT * FROM timezones WHERE timezoneid = 25)) UNION (SELECT 27, 'America/Anchorage' FROM (SELECT 1 AS whocares) AS t WHERE NOT EXISTS (SELECT * FROM timezones WHERE timezoneid = 27)) UNION (SELECT 29, 'America/Adak' FROM (SELECT 1 AS whocares) AS t WHERE NOT EXISTS (SELECT * FROM timezones WHERE timezoneid = 29)) UNION (SELECT 31, 'Pacific/Honolulu' FROM (SELECT 1 AS whocares) AS t WHERE NOT EXISTS (SELECT * FROM timezones WHERE timezoneid = 31))
SELECTin bracketsINSERT INTO t1 (col1, col2) VALUES ( 1, (SELECT id FROM t2 WHERE t2_col1 = 1 LIMIT 1) ), ( 2, (SELECT id FROM t2 WHERE t2_col1 = 2 LIMIT 1) );
31.3.8.10. UPDATE
- Update fields in Multiple Tables
SELECT ... FROM {$from_statement} WHERE {$where_statement} -- eq. to UPDATE Statement UPDATE {$from_statement} SET {$set_statements} WHERE {$where_statement}
UPDATE node, node_revision SET node.comment=1, node_revision.comment = 1 WHERE node_revision.nid = node.nid AND node.type <> 'blog'
Update Joined Tables
UPDATE tableA a JOIN tableB b ON a.a_id = b.a_id JOIN tableC c ON b.b_id = c.b_id SET b.val = a.val+c.val, a.val2 = 'new' WHERE a.val > 10 AND c.val > 10
- Update multiple rows in one query
UPDATE table_users SET cod_user = ( case when user_role = 'student' then '1' when user_role = 'assistant' then '2' when user_role = 'admin' then '3' end), date = 'set to this value for all matched rows' WHERE user_role in ('student', 'assistant', 'admin') AND cod_office = '17389551';
- Can't use subquery in
SET
-- Set row #321's xml column to be the same as row #123 UPDATE a a1 join a a2 ON a2.id=123 SET a1.xml = a2.xml where a1.id=321;
31.3.8.11. LIMIT, OFFSET
LIMIT {[offset,] row_count | row_count OFFSET offset}Can't directly use
LIMITinINsub query This doesn't workSELECT * FROM wp_posts WHERE wp_posts.ID IN ( SELECT p2.ID FROM wp_posts p2 WHERE p2.post_type IN ('posts') LIMIT 5)
Do this
SELECT * FROM wp_posts WHERE wp_posts.ID IN ( SELECT * FROM (SELECT p2.ID FROM wp_posts p2 WHERE p2.post_type IN ('posts') LIMIT 5) AS sq1 )
- if the outter query has
LIMIT, then it will overwrite any subquery'sLIMIT- See 31.3.8.4.1
31.3.8.12. EXISTS
- If only checking existance in other tables but not returning any data from other tables, always use
EXISTSorNOT EXISTS - The following returns nodes that have
field_dealer = 101615andfield_ad_status = activebut not returning any fields from tablesfield_data_*
SELECT count(*) FROM node AS n WHERE n.type='ad_listing' AND n.status = 1 AND EXISTS ( SELECT * FROM field_data_field_dealer d INNER JOIN node nsub ON nsub.nid=d.entity_id AND d.bundle='ad_listing' AND d.field_dealer_target_id IN (101615) WHERE nsub.nid = n.nid ) AND EXISTS ( SELECT * FROM field_data_field_ad_status s INNER JOIN node nsub ON nsub.nid=s.entity_id AND s.bundle='ad_listing' AND s.field_ad_status_value = 'active' WHERE nsub.nid=n.nid )
31.3.8.13. UNION
select_statement_1 UNION [ALL] select_statement_2 UNION [ALL] select_statement_3 ... ...
- In a UNION query, there are at least two SELECT statements
- The 2 SELECT statements must have the same number of columns and the the columns must have compatible data types
- The column headings in each of the SELECT statements do not have to have the same name. The column headings in the result of a
UNIONquery are always taken from the first SELECT statement - If you want to sort the result set of the
UNIONoperation, you can only put anORDER BYclause after the last SELECT statement.ORDER BYclause can't be specified in any other SELECT statements in the UNION query - The column(s) used in
ORDER BYclause can only be taken from the first SELECT statement - If you don't specify an
ORDER BYclause in the UNION query, the result set is always sorted by the first column - If you use
UNION ALL, the entire result set from the second SELECT statement is appended to the first SELECT statement. In this case, there could be duplicate records in the unioned result set If you only use UNION, MySQL removes duplicate rows from the final result set
SELECT s.id AS id, score.score, course.course, null AS certificate FROM student s LEFT JOIN s_score score ON score.sid = s.id LEFT JOIN s_course course ON score.sid = s.id UNION ALL SELECT s.id AS id, null, null, cert.certificate FROM student s LEFT JOIN s_certificate cert ON cert.sid = s.id ORDER BY id
31.3.8.14. CASE, IF(expr1,expr2,expr3), IFNULL(expr1,expr2), NULLIF(expr1,expr2)
- https://dev.mysql.com/doc/refman/5.7/en/control-flow-functions.html
CASE value WHEN [compare_value] THEN result [WHEN [compare_value] THEN result ...] [ELSE result] ENDCASE WHEN [condition] THEN result [WHEN [condition] THEN result ...] [ELSE result] END
SELECT IF(1>2,2,3); -- 3 SELECT IF(1=1,2,3); -- 2 SELECT IF(STRCMP('1', '1') = 0, 2, 3); -- 2 -- IFNULL(expr1, expr2) :: return expr2 if expr1 is null -- NULLIF(expr1, expr2) :: return NULL if expr1 = expr2 -- If value is NULL or another value, then convert and compare in one line SELECT IF(field1 IS NULL OR field1 = '', 'empty', field1) = 'empty' -- Or even shorter SELECT IFNULL( NULLIF( field1, '' ), 'empty' ) = 'empty' SELECT * FROM t1 ORDER BY IF( colA = 'b', colB, NULL ) DESC, colC LIMIT 1; SELECT CASE 1 WHEN 1 THEN 'one' WHEN 2 THEN 'two' ELSE 'more' END AS myColName; -- one SELECT CASE WHEN 1>0 THEN 'true' ELSE 'false' END; -- 'true' SELECT CASE BINARY 'B' WHEN 'a' THEN 1 WHEN 'b' THEN 2 END; -- NULL SELECT * FROM t1 ORDER BY CASE colA WHEN 'value1' THEN colB WHEN 'value2' THEN colC ELSE 'value3' END; SELECT colA, ( SELECT COUNT(*) FROM t2) AS computedCol, CASE computedCol -- Can't use computed column as CASE value!! WHEN 2 then 1 END AS caseCol -- Due to computed columns cannot be used, may need to repeat computed fields SELECT colA, ( SELECT CASE WHEN COUNT(*) > 10 THEN 1 WHEN COUNT(*) > 0 THEN 2 END FROM t2) AS caseCol -- Nested CASE statements SELECT CASE 1 = 1 THEN CASE 2 = 2 THEN 1 ELSE 2 END
31.3.8.15. DISTINCT
- NULL is allowed
- Count non null Distinct values
COUNT(DISTINCT my_expression)
SELECT DISTINCT col1, col2- Equivalent to
GROUP BY
31.3.8.16. Math functions
- All Math Functions
CONV( N, from_base, to_base )- Convert N from from_base to to_base, return a string
- could be a number or a string
- 2 and 36
- Make sure the N and the returned number is less than 2^64, otherwise the result is not accurate
- ROUND(), SUM()
- e.g.
ROUND(SUM(ca.assetvalue),2) - Both return as string in PHP!
- e.g.
- Convert from MySQL string to numeric type in PHP
intval($aColumnValue)- convert to integer
floatval($aColumnValue)convert to float
// no warning var_dump(floatval('') == 0, floatval(' ') == 0, floatval(' 321.32 ') == 321.32, floatval('0.00') == 0, floatval(null) == 0)
- Never do
$convertedValue = $aColumnValue + 0; - throws warning when input value is empty string
- null is fine and converted to 0 in int or float
31.3.8.17. Bitwise
SELECT -- 1 CONV(131072, 10, 2) = 100000000000000000, -- 1 1 << 17 = 131072 -- 1 LOG2(1 << 17) = 17, -- 1 (131072 & (1 << 17)) = 131072, -- 1 (131072 & (1 << 17)) > 0, -- 1 CONV(1 << 17 | 1 << 1, 10, 2) = 100000000000000010, -- 1, have the flag ((1 << 17) | (1 << 1)) & (1 << 1) > 0, -- 1, have the flag (1 << 17 | 1 << 1) & (1 << 1) = (1 << 1), -- 1, doesn't have the flag (1<<17 | 1<<1) & (1<<2) = 0, -- 1, remove a flag CONV( ((1 << 17) | (1 << 1)) & ~ (1 << 1), 10,2) = 100000000000000000, -- 1, remove a flag ((1 << 17) | (1 << 1)) & ~ (1 << 1) = (1 << 17), -- remove no flag (((1 << 17) | (1 << 1)) & ~ 0) = ((1 << 17) | (1 << 1))
31.3.8.18. String functions 31.3.6.5
31.3.8.19. Comparison functions and Operators
COALESCE(value1, value2, ...)- Returns the first non-NULL value in the list, or NULL if there are no non-NULL values
WHERE COALESCE(col1, col2, col3) IS NOT NULLeq.WHERE col1 IS NOT NULL OR col2 IS NOT NULL OR col3 IS NOT NULL
WHERE COALESCE(col1, col2, col3) IS NULLeq.WHERE col1 IS NULL AND col2 IS NULL AND col3 IS NULL
Null-safe equal to operator
-- Like `=` operator. Return 1 if both operands are NULL. Return 0 if one operand is NULL SELECT 1 <=> 1, NULL <=> NULL, 1 <=> NULL; -- 1, 1, 0 SELECT 1 = 1, NULL = NULL, 1 = NULL; -- 1, NULL, NULL
31.3.8.20. Transaction
SET autocommit = 0; -- SET autocommit = OFF; START TRANSACTION; SELECT * FROM abc COMMIT; SET autocommit = 1;
31.3.8.21. Rows to columns
/* Return one row ABC XYZ 123 321 Instead of returning 2 rows ABC 123 XYZ 321 */ SELECT ( SELECT t1.templateid FROM templates t1 WHERE t1.companyid = ? AND t1.templatekey = 'ABC' LIMIT 1 ) AS ABC, ( SELECT t2.templateid FROM templates t2 WHERE tt2.companyid = ? AND tt2.templatekey = 'XYZ' LIMIT 1 ) AS SMS
31.3.9. Admin queries
31.3.9.2. ALTER TABLE
- Alter Table Doc
- column_definition
-- MODIFY eq. MODIFY COLUMN can change definition only MODIFY [COLUMN] col_name column_definition [FIRST | AFTER col_name] column_definition: { data_type [NOT NULL | NULL] [DEFAULT default_value] [AUTO_INCREMENT] [UNIQUE [KEY]] [[PRIMARY] KEY] [COMMENT 'string'] [COLLATE collation_name] [COLUMN_FORMAT {FIXED | DYNAMIC | DEFAULT}] [STORAGE {DISK | MEMORY}] [reference_definition] | data_type [COLLATE collation_name] [GENERATED ALWAYS] AS (expr) [VIRTUAL | STORED] [NOT NULL | NULL] [UNIQUE [KEY]] [[PRIMARY] KEY] [COMMENT 'string'] [reference_definition] } ALTER TABLE `t1` MODIFY `col1` VARCHAR(255); ALTER TABLE `t1` MODIFY `col1` VARCHAR(255) FIRST; -- AFTER can be used -- Duplicate a column ALTER TABLE `t1` ADD `col1duplicate` VARCHAR(255); UPDATE `t1` SET `col1duplicate` = `col1`; -- CHANGE eq. CHANGE COLUMN can change column name -- The order ~FIRST | AFTER anotherColumn~ is optional -- To get column definition, refer to mysql:show create table ALTER TABLE `t1` CHANGE `col1` `col1newname` VARCHAR(255) FIRST; ALTER TABLE `t1` CHANGE COLUMN `col1` `col1` varchar(255) default 'a'; ALTER TABLE `t1` ADD `col1` tinyint DEFAULT 0; -- set a default value ALTER TABLE `t1` ADD `col1` longtext NULL; -- can be null ALTER TABLE `t1` ADD `col1` VARCHAR(255) NULL AFTER `col2`; -- RENAME ALTER TABLE `t1` RENAME COLUMN original_column_name TO new_column_name; -- v8.0+ -- ADD eq. ADD COLUMN ALTER TABLE `t1` ADD COLUMN `col1` VARCHAR(10) DEFAULT NULL; -- default null also means can be null ALTER TABLE `t1` ADD COLUMN `col1` VARCHAR(10) DEFAULT NULL, ADD COLUMN `col2` bigint(20) unsigned DEFAULT NULL, ADD COLUMN `col3` tinyint(1) unsigned DEFAULT 0; -- one query multiple alter's -- DROP column ALTER TABLE `t1` DROP COLUMN `col1`;
31.3.9.3. Duplicate a table
-- to copy indexes and triggers CREATE TABLE new_table LIKE old_table; INSERT INTO new_table SELECT * FROM old_table;
31.3.9.4. Table Status, Table Settings, Table Definition
Filter tables
-- any table with suffix `x` SHOW TABLES LIKE '%x';
- Show table settings for all tables
SHOW TABLE STATUS FROM mydatabase; -- filter by table name SHOW TABLE STATUS WHERE Name LIKE '%users%';
- Engine
- Storage Engine
- Version
- e.g. 10
- Rows
- number of records. MyISAM shows exact while InnoDB shows approximation may vary by as much as 40% to 50%
- Avg_row_length
- avg row size. Refer to mysql:row size
- Collation
- utf8mb4_general_ci
- DDL - Data Definition Language, Table definition
Column definition and index keys
-- Column definition and index keys SHOW CREATE TABLE myTable\G;
- 31.3.9.8
31.3.9.5. Routines
SELECT * FROM INFORMATION_SCHEMA.ROUTINES
31.3.9.6. Deadlock
- TS
Lock wait timeout exceeded; try restarting transaction
-- default lock wait is 50 secs show variables like 'innodb_lock_wait_timeout'; -- see what is locked SHOW ENGINE INNODB STATUS\G; -- Sample output :: https://gist.github.com/levonlee/b816cf8f06fb2d23946661740d356d91
MySQL thread id 19, OS thread handle 140471672895232, query id 209779 localhost 127.0.0.1 root update INSERT INTO yourQuery
SHOW PROCESSLIST; -- eq. to -- SELECT * FROM information_schema.processlist ORDER BY id; -- id is the MySQL thread id e.g. 19 -- May see more info about that thread SELECT * FROM performance_schema.events_statements_history WHERE thread_id = (SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = 19) -- kill this thread meaning to kill the whole connection KILL CONNECTION 19; -- eq. to -- KILL 19; -- To check if it's killed, run SHOW PROCESSLIST;
31.3.9.7. Transpose query results
mysql -u username -p
select * from atable\G
31.3.9.8. Table columns
- To get index keys in a table
- 31.3.9.4.2
- INFORMATION_SCHEMA.COLUMNS
TABLE_NAME Search tables by column name
SELECT DISTINCT TABLE_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_NAME IN ('columnA','ColumnB') AND TABLE_SCHEMA='YourDatabase';
COLUMN_NAME All columns of a table
SELECT column_name, COLUMN_TYPE, COLUMN_DEFAULT, IS_NULLABLE, COLUMN_COMMENT FROM information_schema.columns WHERE table_schema = DATABASE() -- current database AND table_name = 'mytablename'
- e.g.
decimal(10,2) unsigned - e.g.
bigint - e.g.
auto_increment - e.g. 0, 'default string'
- YES or NO
- COLUMN_COMMENT
See ALTER TABLE
alter table myTable modify colA tinyint unsigned default 0 null; ALTER TABLE myTable MODIFY colA {COLUMN_TYPE} DEFAULT {COLUMN_DEFAULT} {IS_NULLABLE}
- Column definition
SHOW FULL COLUMNS
Collation, privileges, comments
SHOW FULL COLUMNS FROM myTable; SHOW FULL COLUMNS FROM myTable WHERE Field = 'col1'; -- only show Type, Null, Key Default, Extra. They're all the same DESCRIBE myTable; EXPLAIN myTable;
31.3.9.9. EXPLAIN
- Use it before
SELECT,DELETE,INSERT,REPLACEANDUPDATEstatements EXPLAIN format=JSON SELECT * FROM aTable- See SHOW FULL PROCESSLIST
- JSON
- query_block
- cost_info
- query_cost
- ordering_operation
- nested_loop
- array
- The sequence matters
- array item
- table
- using_index
- if true, great! True means all the columns needed from the table are found in the INDEX being used.
- False or not exists otherwise
- Same for
GROUP BY
- access_type
- join type. From the best to the worst type
- system
- the table has only one row
- const
at most one matching row. It's used when you compare all parts of a PRIMARY KEY or UNIQUE index and they are constant values.
SELECT * FROM tbl_name WHERE primary_key_part1=1 AND primary_key_part2=2;
- eq_ref
one row is read from this table for each combination of rows from the previous tables. It's used when all parts of an index are used by the join and the index is a PRIMARY KEY or UNIQUE NOT NULL index. Operator
=SELECT * FROM ref_table,other_table WHERE ref_table.key_column = other_table.column; SELECT * FROM ref_table,other_table WHERE ref_table.key_column_part1 = other_table.column AND ref_table.key_column_part2 = 1;
- ref
- all rows with matching index values are read from this table for each combo of rows from the previous tables.
Examples
SELECT * FROM ref_table WHERE key_column= expr; SELECT * FROM ref_table,other_table WHERE ref_table.key_column = other_table.column; SELECT * FROM ref_table,other_table WHERE ref_table.key_column_part1 = other_table.column AND ref_table.key_column_part2 = 1;
- It's used if the join uses only a leftmost prefix of the key or if the key is not a PRIMARY KEY or UNIQUE index (in other words, if the join cannot select a single row based on the key value). If the used key that matches only a few rows, this is a good join type.
- Operator:
=or<=>
- Operator:
- fulltext
- join using
FULLTEXTindex - (no term)
- ref_or_null
- (no term)
- index_merge
- (no term)
- unique_subquery
- (no term)
- index_subquery
- (no term)
- range
- (no term)
- index
- ALL
- a full table scan is done
- (no term)
- filtered
- 100
- bad. Means no filtering of rows occured
- (no term)
- key
- The index that is actually used
- no index is used
- (no term)
- cost_info
- (no term)
- materialized_from_subquery
- query_block
- cost_info
- grouping_operation
- table
- query_block
- table
- cost_info
- query_block
- Non JSON
- Output
- https://dev.mysql.com/doc/refman/8.0/en/explain-output.html
- type
- same as JSON access_type
- operator
=- (no term)
- key
- null
- when it's NULL means it's a full table scan
- Or the target index key is not used due to the provided value (from previous tables) does not exist in this table. Extra column may show Impossible ON condition
- rows
- number of rows to scan
- when
keyis NULL, it'll be the table's total number of rows
- when
31.3.9.10. SHOW FULL PROCESSLIST
- https://dev.mysql.com/doc/refman/8.0/en/thread-information.html
- Response columns
- Id
- this is the connection ID
- User
- e.g. root
- Host
- IP:port
- db
- Command
- https://dev.mysql.com/doc/refman/8.0/en/thread-commands.html
- Time
- number of seconds the thread has been in its current state
- State
- https://dev.mysql.com/doc/refman/8.0/en/general-thread-states.html
mysqladmin -uroot -pABC -h abc.com --sleep=1 --count=5 --verbose processlistprocesslist- command
--verbose, -v- eq.
SHOW FULL PROCESSLIST --sleep=delayInSeconds, -i delayInSeconds- run the command repeatedly
--count=N, -c N- number of iterations to run
31.3.9.11. SIGNAL / RESIGNAL throw error or warning
- https://dev.mysql.com/doc/refman/5.7/en/signal.html
- No priveleges required
- Inside a
BEGIN ... ENDblock
- Basics
SIGNAL condition_value [SET signal_information_item [, signal_information_item] ...] condition_value: { SQLSTATE [VALUE] sqlstate_value | condition_name } signal_information_item: condition_information_item_name = simple_value_specification condition_information_item_name: { CLASS_ORIGIN | SUBCLASS_ORIGIN | MESSAGE_TEXT | MYSQL_ERRNO | CONSTRAINT_CATALOG | CONSTRAINT_SCHEMA | CONSTRAINT_NAME | CATALOG_NAME | SCHEMA_NAME | TABLE_NAME | COLUMN_NAME | CURSOR_NAME }SIGNAL SQLSTATE '01000' SET MESSAGE_TEXT = 'my warning';
CREATE PROCEDURE p (pval INT) BEGIN DECLARE specialty CONDITION FOR SQLSTATE '45000'; IF pval = 0 THEN SIGNAL SQLSTATE '01000'; -- '01' :: warning class, which does not terminate ELSEIF pval = 1 THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'An error occurred'; ELSEIF pval = 2 THEN SIGNAL specialty SET MESSAGE_TEXT = 'An error occurred'; ELSE SIGNAL SQLSTATE '01000' SET MESSAGE_TEXT = 'A warning occurred', MYSQL_ERRNO = 1000; -- The last SIGNAL replaces MESSAGE_TEXT AND MYSQL_ERRNO set previously SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'An error occurred', MYSQL_ERRNO = 1001; END IF; END;
- SQLSTATE
- 5-character string
- First 2 characters are called class
- generic value, exception, means "unhandled user-defined exception"
- Overrides MYSQL_ERRNO
- Classes
- '00'
- success. Do not use this in
SIGNAL/RESIGNAL - '01'
- warning but not failing the statement
- '02'
- not found. By default, it will work just as warning: not failing the statement
- > '02'
- exception, fails the statement
- Classes
- 5-character string
- RESIGNAL
- Pass upper level warning/error onto the current level
- Must be inside a 31.3.9.12
DELIMITER $$ CREATE PROCEDURE Divide(IN numerator INT, IN denominator INT, OUT result double) BEGIN DECLARE division_by_zero CONDITION FOR SQLSTATE '22012'; DECLARE CONTINUE HANDLER FOR division_by_zero RESIGNAL SET MESSAGE_TEXT = 'Division by zero / Denominator cannot be zero'; -- IF denominator = 0 THEN SIGNAL division_by_zero; ELSE SET result := numerator / denominator; END IF; END
31.3.9.12. Handler
- https://dev.mysql.com/doc/refman/8.0/en/declare-handler.html
- Handler declarations must appear after variable or condition declarations
- condition_value
- SQLWARNING
- Shorthand for the class of SQLSTATE values that begin with '01'
- NOT FOUND
- Shorthand for the class of SQLSTATE values that begin with '02'
- SQLEXCEPTION
- Shorthand for the class of SQLSTATE values that do not begin with '00', '01', or '02'
DECLARE handler_action HANDLER FOR condition_value [, condition_value] ... statement handler_action: { CONTINUE | EXIT | UNDO } condition_value: { mysql_error_code | SQLSTATE [VALUE] sqlstate_value | condition_name | SQLWARNING | NOT FOUND | SQLEXCEPTION }
31.3.9.13. Profiling
SET profiling = 1; -- Run query #1 -- Run query #2 SHOW PROFILES; -- Query_ID and Duration
31.3.9.14. Row size and number of columns
- https://dev.mysql.com/doc/mysql-reslimits-excerpt/5.7/en/column-count-limit.html
- Row size
- Hard limit
- 65,535 bytes (slightly less than 64KB)
- (no term)
- mysql:datatype:blob only contributes 9 bytes
- (no term)
- mysql:datatype:text only contributes 12 bytes
- (no term)
- mysql:innodb
- See a table's avg row size
- mysql:avg_row_length
- Column Count
- Hard limit
- 4096
- (no term)
- Determined by row size
- mysql:innodb
- 1017
- including virutal generated columns
31.3.10. User-Defined Variables
- https://dev.mysql.com/doc/refman/8.0/en/user-variables.html
They can still be defined in non-
SETstatements up to MySQL v8, but in the future will be removedSELECT @myVariable := myTable.col1 AS myAlias FROM myTable
SET @s3 = 'http://s3.amazon.com/abc/'; -- eq. to SET @s3 := 'http://s3.amazon.com/abc/'; SET @s1 = 'da', @s2 = 'ba'; -- variable name is alphanumeric with `.`, `_` and `$` -- @'my-var', @"my-var", @`my-var` -- Case-insensitive, up to 64 characters -- integer, decimal, floating-point, binary or nonbinary string, or NULL SELECT @s3;
31.3.11. Routines, Procedures, User Defined Function
- Stored Procedure and Stored Functions are stored under the schema as routines
- function returns a value
db_name.sp_nameList all routines
SELECT * FROM INFORMATION_SCHEMA.ROUTINES WHERE 0=0 -- AND ROUTINE_TYPE = 'FUNCTION'; -- AND ROUTINE_TYPE = 'PROCEDURE'; -- AND ROUTINE_NAME = 'myname';
31.3.11.1. Procedure
CREATE [DEFINER = user] PROCEDURE [IF NOT EXISTS] sp_name ([proc_parameter[,...]]) [characteristic ...] routine_body
- proc_parameter
- Types
- IN
- the procedure might modify but the caller cannot see the modification
- OUT
- the value is passed back from the procedure back to the caller. Initial value is
NULL - (no term)
- INOUT
- Types
DELIMITER // CREATE PROCEDURE GetAllProducts() BEGIN SELECT * FROM products; END // DELIMITER ; -- CALL GetAllProducts(); DELIMITER // CREATE PROCEDURE GetAllProducts(IN country CHAR(3), OUT cities INT) BEGIN SELECT COUNT(*) INTO cities FROM products WHERE CountryCode = country;; END // DELIMITER ; -- CALL GetAllProducts('JPN', @cities); -- SELECT @cities;
31.3.11.2. Characteristics
characteristic:
COMMENT 'string'
| LANGUAGE SQL
| [NOT] DETERMINISTIC
| { CONTAINS SQL | NO SQL | READS SQL DATA | MODIFIES SQL DATA }
| SQL SECURITY { DEFINER | INVOKER }
Don't specify LANGUAGE SQL
A routine is considered “deterministic” if it always produces the same result for the same input parameters, and “not deterministic” otherwise. If neither DETERMINISTIC nor NOT DETERMINISTIC is given in the routine definition, the default is NOT DETERMINISTIC. To declare that a function is deterministic, you must specify DETERMINISTIC explicitly.
Non-deterministic examples are data modification, have UPDATE, INSERT or DELETE statement(s).
Default without SET GLOBAL log_bin_trust_function_creators = 1; is NOT DETERMINISTIC
Without specifying DETERMINISTIC and with the default setting, this error throws:
This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)
{ CONTAINS SQL | NO SQL | READS SQL DATA | MODIFIES SQL DATA }
- CONTAINS SQL Default, indicates that the routine does not contain statements that read or write data. Examples of such statements are SET @x = 1 or DO RELEASE_LOCK('abc'), which execute but neither read nor write data.
- indicates that the routine contains no SQL statements.
- DATA indicates that the routine contains statements that read data (for example, SELECT), but not statements that write data.
- indicates that the routine contains statements that may write data (for example, INSERT or DELETE).
The parameter list enclosed within parentheses must always be present. If there are no parameters, an empty parameter list of () should be used. Parameter names are not case sensitive.
31.3.11.3. Update definer
update mysql.proc set definer='root@localhost' where name = 'myprocedurename'; -- For MySQL v8.0, there's no mysql.proc table
31.3.11.4. Stored Procedure
- Syntax
- Refer to mysql:routine:characteristics
CREATE [DEFINER = { user | CURRENT_USER }] PROCEDURE sp_name ([proc_parameter[,...]]) [characteristic ...] routine_body proc_parameter: [ IN | OUT | INOUT ] param_name type- IN
- Default. It passes a value into a procedure. The procedure might modify the value, but the modification is not visible to the caller when the procedure returns.
- OUT
- passes a value from the procedure back to the caller. Its initial value is NULL within the procedure, and its value is visible to the caller when the procedure returns. An INOUT parameter is initialized by the caller, can be modified by the procedure, and any change made by the procedure is visible to the caller when the procedure returns.
- (no term)
- Use
()if there is no parameter - (no term)
- In body, stored function can be used
For each OUT or INOUT parameter, pass a user-defined variable in the CALL statement that invokes the procedure so that you can obtain its value when the procedure returns. If you are calling the procedure from within another stored procedure or function, you can also pass a routine parameter or local routine variable as an IN or INOUT parameter.
Routine parameters cannot be referenced in statements prepared within the routine
Each SELECT statement that does not insert into a table or a variable will produce a result set.
mysql> delimiter // mysql> CREATE PROCEDURE simpleproc (OUT param1 INT) -> BEGIN -> SELECT COUNT(*) INTO param1 FROM t; -> END; // Query OK, 0 rows affected (0.00 sec) mysql> delimiter ; mysql> CALL simpleproc(@a); Query OK, 0 rows affected (0.00 sec) mysql> SELECT @a; +------+ | @a | +------+ | 3 | +------+ 1 row in set (0.00 sec) - Prepare Statement, Convert delimited string into temp table
A prepare statement is unique to a session. If it's not deallocated, it will be deallocated automatically when the session ends. mysql:string:list:temp table
DROP PROCEDURE IF EXISTS `getArticlesByCats`; CREATE PROCEDURE getArticlesByCats(IN pDelim VARCHAR(32), IN pStr TEXT ) DETERMINISTIC MODIFIES SQL DATA BEGIN DROP TABLE IF EXISTS li_temp_explode; CREATE TEMPORARY TABLE li_temp_explode (id INT AUTO_INCREMENT PRIMARY KEY NOT NULL, word VARCHAR(40)); SET @sql := CONCAT('INSERT INTO li_temp_explode (word) VALUES (', REPLACE(QUOTE(pStr), pDelim, '\'), (\''), ')'); PREPARE myStmt FROM @sql; EXECUTE myStmt; DEALLOCATE PREPARE myStmt; -- Later use the temp table in IN statement SELECT * FROM node n WHERE EXISTS( SELECT * FROM cats c WHERE c.cid IN ( SELECT CAST(word AS UNSIGNED) FROM li_temp_explode ) ) END;
word column is string
CALL `getArticlesByCats`(',', '1,31,5,6,7,8,9,10'); SELECT * FROM li_temp_explode;
- List all stored procedures of all dbs that the user has access to
SHOW PROCEDURE STATUS;- List stored procedures of a db
SHOW PROCEDURE STATUS WHERE db = 'mydbname'- List stored procedures by name
SHOW PROCEDURE STATUS WHERE name LIKE '%product%'- Show the code of a stored procedure
SHOW PROCEDURE my_stored_procedure_name- Drop a stored procedure
DROP PROCEDURE IF EXISTS my_sp_name
31.3.11.5. User Defined Function mysql:udf
- UDF always returns a value
- Syntax
- Refer to [[mysql:routine:characteristics][mysql:routine:characteristics
CREATE [DEFINER = { user | CURRENT_USER }] FUNCTION sp_name ([func_parameter[,...]]) RETURNS type [characteristic ...] routine_body func_parameter: param_name type type: Any valid MySQL data typeSET @lis3 = 'http://s3.amazonaws.com/abc/'; SET @lifs = 'http://abc.com/sites/default/files/'; SET @litt = 'http://abc.com/'; SET @lir1 = '^(s3://)(.*)'; SET @lir2 = CONCAT(@lis3, '\\2'); SET @lir3 = '^(public://)(.*)'; SET @lir4 = CONCAT(@lifs, '\\2'); DROP FUNCTION IF EXISTS `liFileUri`; CREATE FUNCTION `liFileUri` (`uri` VARCHAR(255)) RETURNS VARCHAR(255) DETERMINISTIC NO SQL BEGIN RETURN REGEXP_REPLACE(REGEXP_REPLACE(uri, @lir1, @lir2), @lir3, @lir4); END; DROP FUNCTION IF EXISTS `liConCat2`; CREATE FUNCTION `liConCat2`(`v1` TEXT, `v2` TEXT) RETURNS TEXT DETERMINISTIC NO SQL BEGIN DECLARE s TEXT; IF (v1 IS NULL AND v2 IS NULL) THEN SET s = NULL; ELSE SET s = CONCAT_WS(0x1F, v1, v2); END IF; RETURN s; END; -- UDF can't have optional parameter. e.g. have to create liConCat3 to concat 3 fields DROP FUNCTION IF EXISTS `liFixNull`; CREATE FUNCTION `liFixNull`(`text` TEXT) RETURNS TEXT DETERMINISTIC BEGIN -- Use this with CONCAT('aString', liFixNull(aColumn)) is useful because CONCAT returns NULL if any is null RETURN CASE text WHEN 'NULL' THEN NULL WHEN '' THEN NULL ELSE text END; END;
- Return a single value from SELECT statement
DROP FUNCTION IF EXISTS udfreturnselect; DELIMITER ;; CREATE FUNCTION udfreturnselect(parentQueryT1Col1_ BIGINT(20), parentQueryT2Col1_ DECIMAL(20, 2)) RETURNS INT(11) DETERMINISTIC CONTAINS SQL BEGIN DECLARE udfreturnselect_result INT; SELECT CASE WHEN (IFNULL(parentQueryT2Col1_, 0) - SUM(IFNULL(t3.invamount, 0)) < 0.01 AND parentQueryT2Col1_ > 0) THEN 1 WHEN (IFNULL(parentQueryT2Col1_, 0) - SUM(IFNULL(t3.invamount, 0)) > 0 AND SUM(IFNULL(t3.invamount, 0)) > 0) THEN 2 ELSE 3 END INTO udfreturnselect_result FROM t3 WHERE t3.roundid = parentQueryT1Col1_ AND t3.replaced = 0; RETURN udfreturnselect_result; END ;; DELIMITER ;
31.3.12. SQL Fiddle
- http://sqlfiddle.com/
- MySQL 5.6
- https://dbfiddle.uk/
- MySQL 8
Schema
CREATE TABLE student ( `sid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, `firstname` VARCHAR(20) NOT NULL, `lastname` VARCHAR(20) NOT NULL, `reg_date` TIMESTAMP ); INSERT INTO student ( firstname, lastname, reg_date ) VALUES ( 'first1', 'last1', current_timestamp), -- 1 ( 'first2', 'last2', current_timestamp), -- 2 ( 'first3', 'last3', current_timestamp); -- 3 CREATE TABLE course ( `cid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, `coursename` VARCHAR(20) NOT NULL, `semaster_date` BIGINT(20) UNSIGNED NOT NULL ); INSERT INTO course ( coursename, semaster_date ) VALUES ( 'Math', 0 ), -- 1 ( 'English', 0 ), -- 2 ( 'PHP', 1593015426 ); -- 3 CREATE TABLE s_course ( `scid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, `sid` INT(6) UNSIGNED NOT NULL, `cid` INT(6) UNSIGNED NOT NULL ); INSERT INTO s_course ( sid, cid ) VALUES ( 1, 1 ), -- 1 ( 1, 2 ), -- 2 ( 2, 3 ); -- 3 CREATE TABLE s_score ( `ssid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, `scid` INT(6) UNSIGNED NOT NULL, `score` INT(3) UNSIGNED NOT NULL, `midterm_score` INT(3) UNSIGNED NOT NULL DEFAULT 0, `assignment_score` INT(3) UNSIGNED NOT NULL DEFAULT 0, `final_exam_score` INT(3) UNSIGNED NULL DEFAULT 0, `test_date` BIGINT(20) UNSIGNED NOT NULL ); INSERT INTO s_score ( scid, score, midterm_score, assignment_score, final_exam_score, test_date ) VALUES ( 1, 20, 21, 22, 23, 0 ), -- Student 1 ( 2, 30, 31, 32, 33, 0 ), -- 2 ( 3, 40, 41, 42, 43, 0 ), -- 3 ( 3, 40, 41, 42, null, 0 ), -- 3 ( 1, 50, 51, 52, 53, 1593015426 ), -- 1 ( 1, 60, 61, 62, 63, 1594015426 ), -- 1 ( 2, 70, 71, 72, 73, 1595015426 ), -- 2 ( 1, 60, 61, 62, 63, 1594015426 ); -- 1
31.3.13. MySQLTutorial.org
31.3.14. Dump, Restore
mysqldump -u user -p database_name > database_name_backup.sql # `--single-transaction` prevents table from being locked mysqldump --single-transaction --quick --max_allowed_packet=512M --compress --routines --triggers --events -uuser -p database_name > backup.sql # Assuming there's enough disk space mysqldump -uuser -p database_name > backup.sql && gzip backup.sql # Remove encryption sed -i "s/ENCRYPTION='Y'//g" backup.sql # Not enough disk space then use pipe mysqldump -uroot -p database_name | gzip > backup.sql.gz # dump specific tables mysqldump -uroot -pMyPassword --single-transaction --quick --compress my_database t1 t2 t3 | gzip > a.sql.gz # No `CREATE DATABASE` # don't dump some tables mysqldump -uroot -p --ignore-table=database_name.tableNameToIgnore database_name # Use config file to avoid typing password mysqldump --defaults-extra-file=/path/to/.mydb.cnf -u user db_name > db_name_bk.sql # without any -u or -p, the user config file `~/.my.cnf` is used. Refer to mariadb:config mysqldump db_name > db_name.sql # dump a table mysqldump db_name table_name > table_name.sql # restore one or more tables mysql -u username -p db_name < /path/to/table_name.sql zcat 1.sql.gz | mysql -uroot -p my_db # Change a table name in mysqldump file # /*!40000 ALTER TABLE `my_table` ENABLE KEYS */; (_oldname="my_table"; _newname="my_table_new"; cat 1.sql \ | sed "s/^CREATE TABLE \`${_oldname}\`/CREATE TABLE \`${_newname}\`/g" \ | sed "s/^DROP TABLE IF EXISTS \`${_oldname}\`/DROP TABLE IF EXISTS \`${_newname}\`/g" \ | sed "s/^INSERT INTO \`${_oldname}\`/INSERT INTO \`${_newname}\`/g" \ | sed "s/^-- Table structure for table \`${_oldname}\`/-- Table structure for table \`${_newname}\`/g" \ | sed "s/^-- Dumping data for table \`${_oldname}\`/-- Dumping data for table \`${_newname}\`/g" \ | sed "s/^LOCK TABLES \`${_oldname}\`/LOCK TABLES \`${_newname}\`/g" \ | sed "s/^\/\*\!40000 ALTER TABLE \`${_oldname}\`/\/\*\!40000 ALTER TABLE \`${_newname}\`/g" \ | grep --color "${_oldname}") (_oldname="my_table"; _newname="my_table_new"; gunzip < 1.sql.gz \ | sed "s/^CREATE TABLE \`${_oldname}\`/CREATE TABLE \`${_newname}\`/g" \ | sed "s/^DROP TABLE IF EXISTS \`${_oldname}\`/DROP TABLE IF EXISTS \`${_newname}\`/g" \ | sed "s/^INSERT INTO \`${_oldname}\`/INSERT INTO \`${_newname}\`/g" \ | sed "s/^-- Table structure for table \`${_oldname}\`/-- Table structure for table \`${_newname}\`/g" \ | sed "s/^-- Dumping data for table \`${_oldname}\`/-- Dumping data for table \`${_newname}\`/g" \ | sed "s/^LOCK TABLES \`${_oldname}\`/LOCK TABLES \`${_newname}\`/g" \ | sed "s/^\/\*\!40000 ALTER TABLE \`${_oldname}\`/\/\*\!40000 ALTER TABLE \`${_newname}\`/g" \ | mysql -uroot -p my_db)
# Change a table name in mysqldump file # /*!40000 ALTER TABLE `my_table` ENABLE KEYS */; (_oldname="my_table"; _newname="my_table_new"; cat 1.sql \ | sed "s/^CREATE TABLE \`${_oldname}\`/CREATE TABLE \`${_newname}\`/g" \ | sed "s/^DROP TABLE IF EXISTS \`${_oldname}\`/DROP TABLE IF EXISTS \`${_newname}\`/g" \ | sed "s/^INSERT INTO \`${_oldname}\`/INSERT INTO \`${_newname}\`/g" \ | sed "s/^-- Table structure for table \`${_oldname}\`/-- Table structure for table \`${_newname}\`/g" \ | sed "s/^-- Dumping data for table \`${_oldname}\`/-- Dumping data for table \`${_newname}\`/g" \ | sed "s/^LOCK TABLES \`${_oldname}\`/LOCK TABLES \`${_newname}\`/g" \ | sed "s/^\/\*\!40000 ALTER TABLE \`${_oldname}\`/\/\*\!40000 ALTER TABLE \`${_newname}\`/g" \ | grep --color "${_oldname}")
.mydb.cnf
[mysqldump] password='enclosedwithdoublequotes'
MySQL is inside a container some-mysql and dump it on host
docker exec some-mysql sh -c 'exec mysqldump --all-databases -uroot -p"your-passowrd"' > /some/path/on/your/host/all-databases.sql
Restore
cat backup.sql | docker exec -i CONTAINER /usr/bin/mysql -u root --password=root DATABASE
Dump a remote database
mysqldump -P3306 -h192.168.20.151 -u root -p databasename > c:/my.sql
31.3.14.1. Dump data
-- MySQL dump 10.13 Distrib 5.7.37, for Linux (x86_64) -- -- Host: localhost Database: loanstudio -- ------------------------------------------------------ -- Server version 5.7.37 /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */; /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */; /*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */; /*!40101 SET NAMES utf8 */; /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */; /*!40103 SET TIME_ZONE='+00:00' */; /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; -- -- Table structure for table `abc` -- DROP TABLE IF EXISTS `abc`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `abc` ( -- some columns ) ENGINE=InnoDB AUTO_INCREMENT=341 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `lawyerintegrationbills` -- LOCK TABLES `abc` WRITE; /*!40000 ALTER TABLE `abc` DISABLE KEYS */; INSERT INTO `abc` VALUES ( -- row ) -- rows -- ,() ; /*!40000 ALTER TABLE `abc` ENABLE KEYS */; UNLOCK TABLES; -- repeat for tables -- -- Dumping events for database 'mydb' -- /*!50003 SET sql_mode = @saved_sql_mode */ ; /*!50003 SET character_set_client = @saved_cs_client */ ; /*!50003 SET character_set_results = @saved_cs_results */ ; /*!50003 SET collation_connection = @saved_col_connection */ ; /*!50003 DROP FUNCTION IF EXISTS `myfuncs` */; /*!50003 SET @saved_cs_client = @@character_set_client */ ; /*!50003 SET @saved_cs_results = @@character_set_results */ ; /*!50003 SET @saved_col_connection = @@collation_connection */ ; /*!50003 SET character_set_client = latin1 */ ; /*!50003 SET character_set_results = latin1 */ ; /*!50003 SET collation_connection = latin1_swedish_ci */ ; /*!50003 SET @saved_sql_mode = @@sql_mode */ ; /*!50003 SET sql_mode = 'NO_ENGINE_SUBSTITUTION' */ ; DELIMITER ;; CREATE DEFINER=`root`@`localhost` FUNCTION `myfunceffstatus`(date_ bigint, status_ int, intadj_ bigint, dischargedate_ bigint, effdate_ bigint) RETURNS int(11) DETERMINISTIC begin if status_=0 or status_=3 then return status_; end if; if status_=1 then if date_<intadj_ then return 0; end if; if date_>effdate_ then return 4; end if; return 1; end if; if status_=2 then if date_ <intadj_ then return 0; end if; if date_<dischargedate_ then return 1; end if; return 2; end if; end ;; DELIMITER ; -- repeat for routines/functions /*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */; /*!40101 SET SQL_MODE=@OLD_SQL_MODE */; /*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */; /*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */; /*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */; /*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */; /*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */; /*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */; -- Dump completed on 2022-04-15 10:53:06
31.3.14.2. Dump table definition only
# all tables in a db mysqldump -uroot -p --no-data mydatabase > my-all-tables-def.sql # for a table only mysqldump -uroot -p --no-data mydatabase mytable1 > mytable1-def.sql # --no-create-info dump data not definition
31.3.14.3. Dump query result as CSV
SELECT ... INTO OUTFILE
- It's for the purpose of
LOAD DATA. It has its limitation- Does not work for field value that is multiple lines
- It requires special permissions otherwise it will throw this error
- ERROR 1045 (28000) at line 1: Access denied for user 'root'@'%' (using password: YES)
SELECT * FROM a INTO OUTFILE '/var/lib/mysql-files/a.txt' -- default ESCAPED BY is '\', disable escape: ESCAPED BY '' FIELDS TERMINATED BY ',' ENCLOSED BY '"' -- place ESCAPED BY here before LINES LINES TERMINATED BY '\n'
mysql -e "SELECT * FROM a INTO OUTFILE '/var/lib/mysql-files/a.txt' FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\\n';" -u root -p abc-- need to use this directory SHOW VARIABLES LIKE "secure_file_priv";
- It's for the purpose of
mysql -e "SELECT * FROM t1 ORDER BY id DESC LIMIT 1\G" -uroot -pABC databasename > /tmp/lili.tsv
- Dump as tsv (tab delimited)
Script file can be used instead of
-emysql -uroot -pABC databasename < script.sql > /tmp/lili.tsv
31.3.14.4. Dump a row as INSERT statements
# --no-create-info, -t :: no CREATE TABLE statements # --where='where_condition', -w 'where_condition' # --complete-insert :: add field names in the INSERT statement # --compact :: remove comments. This enables --skip-add-drop-table, --skip-add-locks, --skip-comments, --skip-disable-keys, and --skip-set-charset # --extended-insert, -e :: INSERT statements using multiple-row syntax that includes several VALUES lists # --set-gtid-purged=OFF :: just to make sure only INSERT statements are returned mysqldump myDB myTable --no-create-info --compact --complete-insert --set-gtid-purged=OFF --where='primaryCol=1'
31.3.14.5. Restore a table
# create a backup mysqldump -uroot -p mydb mytable > mytable.sql mysql -uroot -p mydb < mytable.sql # From a full database backup # Method 1: From a full database backup, create a temporary db # - make sure dbBackup.sql does not have `use mydb`. phpMyAdmin might have it but mysqldump won't have it mysql -uroot -p -e "create database temporary" mysql -uroot -p temporary < dbBackup.sql mysqldump -uroot -p temporary mytable > mytable.sql mysql -uroot -p mydb < mytable.sql mysql -uroot -p -e "drop database temporary" # Method 2: From a full database backup, extract a table from a db backup sed -n -e '/-- Table structure for table `abc`/,/UNLOCK TABLES/p' dbBackup.sql > mytable.sql
31.3.14.6. Release space
# See what tables without row data are dumped mysqldump --single-transaction --routines --triggers --events -uroot -p --all-databases > backup.sql # --all-databases, -A :: `mysql` DB is exported but not `performance_schema` # --databases, -B :: `-B mydb1 mydb2` # Don't export `mysql` nor `performance_schema` DB's mysqldump --single-transaction --routines --triggers --events -uroot -p -B mydb1 mydb2 > backup.sql sed -i "s/ENCRYPTION='Y'//g" backup.sql mysqladmin -uroot -p drop mydb1 drop mydb2 mysql -uroot -p -e 'SET GLOBAL innodb_fast_shutdown = 0' mysqladmin shutdown rm -f /var/lib/mysql/ib{data1,_logfile*} service mysql start mysqladmin -uroot -p create mydb1 mysqladmin -uroot -p create mydb2 mysql -uroot -p < backup.sql
31.3.15. Binary Log
https://dev.mysql.com/doc/internals/en/binary-log-overview.html
See if it's enabled and see if the format :: ROW, MIXED or STATEMENT
select variable_value as "BINARY LOGGING STATUS (log-bin) :: " from information_schema.global_variables where variable_name='log_bin'; select variable_value as "BINARY LOG FORMAT (binlog_format) :: " from information_schema.global_variables where variable_name='binlog_format' // show variables like 'datadir';
To update the setup, you need to update the MySQL configuration file (my.ini or my.cnf) by adding the following lines to the [mysqld] section (the Server Section).
log-bin=bin.log log-bin-index=bin-log.index max_binlog_size=100M binlog_format=row socket=mysql.sock
show variables like 'datadir'; // Log file location :: datadir/bin.0001.log show binary logs;
https://jinyuwang.weebly.com/for-mysql/how-to-enable-binary-logging-for-mysql
Recover from deleted rows
mysqlbinlog binary_log_file > query_log.sql
Then search for missing rows.
31.3.16. phpMyAdmin
https://www.phpmyadmin.net/downloads/ Just put sql-admin to any Wordpress website root directory and run http://yourwebsiteip:port/sql-admin You don't need to do anything else.
It first loads sql-admin\libraries\config.default.php and then load sql-admin\config.inc.php To create a config.inc.php, just duplicate sql-admin/config.sample.inc.php
/* Authentication type */ // Use cookie //$cfg['Servers'][$i]['auth_type'] = 'cookie'; // If encounter "Failed to set session cookie. Maybe you are using HTTP instead of HTTPS", use http $cfg['Servers'][$i]['auth_type'] = 'http'; /* Server parameters */ // change host $cfg['Servers'][$i]['host'] = 'database'; $cfg['Servers'][$i]['compress'] = false; $cfg['Servers'][$i]['AllowNoPassword'] = false;
31.3.17. Adminer db:adminer
When choose to dump (export in newer version) a database, leave blank for Database (don't DROP+CREATE) but DROP+CREATE for tables
Routines and Events for Database Only Triggers for Tables Data :: INSERT
-- START We don't want these lines DROP DATABASE IF EXISTS `your_db`; CREATE DATABASE `your_db` USE `your_db` -- END We don't want these lines DROP TABLE IF EXISTS `wp_abc`; CREATE TABLE `wp_ABC` ...
31.3.18. UC: Concatenate values of a column across multiple rows about the same record
- mysqlfiddle
Syntax
GROUP_CONCAT([DISTINCT] expr [,expr ...] [ORDER BY {unsigned_integer | col_name | expr} [ASC | DESC] [,col_name ...]] [SEPARATOR str_val])
- Without SEPARATOR, default is comma
,GROUP_CONCAT(col1)
GROUP_CONCATonly concatenates non-null values. If all values are null, null will be returnedGROUP_CONCATby default only handles 1024 letters. Change it for the current sessionSET SESSION group_concat_max_len = 65535;
SELECT s.id, s.firstname, s.lastname , GROUP_CONCAT(DISTINCT ts.score ORDER BY ts.testDate SEPARATOR ', ') AS student_scores , GROUP_CONCAT(DISTINCT liConCat2(c.course, c.year) SEPARATOR 0x1E) AS courses FROM student s LEFT JOIN s_score ts ON ts.sid = s.id LEFT JOIN s_course c ON c.sid = s.id GROUP BY s.id ORDER BY s.lastname
If there're multiple fields that have multiple values, e.g. a student can have multiple scores and multiple courses, and they are joined,
Total number of rows are for one student is: N(courses of student) X N(scores of student), N(x) is number of x, when x is zero, N(0) = 1.
When you do GROUP_CONCAT on course, use DISTINCT inside.
Concatenate multiple fields
GROUP_CONCAT(DISTINCT liConCat2(c.course, c.year) SEPARATOR 0x1E) AS courses
31.3.18.1. Concatenate strings, use a different separator
- 0x1F (31): unit (fields) separator
- 0x1E (30): record separator
- 0x1D (29): group separator
GROUP_CONCAT(foo SEPARATOR 0x1D) $array = explode(chr(29),$s);
31.3.18.2. JSON_ARRAYAGG
- Return a string of JSON or NULL
- Pros
- No length limit like 31.3.18
- Cons
- Since MySQL 5.7.22
DISTINCTis not allowed and other expressions are allowed
SELECT s.sid, s.firstname, s.lastname , JSON_ARRAYAGG(ts.final_exam_score) AS student_scores FROM student s LEFT JOIN s_score ts ON ts.ssid = s.sid LEFT JOIN s_course c ON c.sid = s.sid GROUP BY s.sid ORDER BY s.lastname
31.3.19. UC: Split Comma-Separated Values mysql:split list
Given table dashboards
| ID | recipients |
|---|---|
| 1 | a@x.ca,b@x.ca |
| 2 | b@x.ca |
| 3 | c@x.ca |
Result
| count | |
|---|---|
| b@x.ca | 2 |
| a@x.ca | 1 |
| c@x.ca | 1 |
#+NAME Create a temporary table
-- create X rows which is equal or bigger than the max number of items across every `recipients` column. -- e.g. row 1 has 2 items, row 2 and 3 has 1 item -- Max number of items across all rows is then 2 -- A temporary table will have a column named `n` and it has 2 rows create temporary table numbers as ( select 1 as n union select 2 as n union select 3 as n -- ... )
A temporary table numbers is created with 2 rows
| n |
|---|
| 1 |
| 2 |
SELECT * JOIN dashboards JOIN numbers ON char_length(recipients) - char_length(replace(recipients, ',', '')) >= numbers.n - 1 -- ON (number of commas in dashboards.recipients) < (n - 1)
Result
| ID | recipients | n |
|---|---|---|
| 1 | a@x.ca,b@x.ca | 1 |
| 1 | a@x.ca,b@x.ca | 2 |
| 2 | b@x.ca | 1 |
| 3 | c@x.ca | 1 |
SELECT id, substring_index( substring_index(recipients, ',', n), ',', -1 ) AS email FROM dashboards JOIN numbers ON char_length(recipients) - char_length(replace(recipients, ',', '')) >= n - 1
Result
| ID | |
|---|---|
| 1 | a@x.ca |
| 1 | b@x.ca |
| 2 | b@x.ca |
| 3 | c@x.ca |
All together
SELECT email, count(1) FROM ( SELECT id, substring_index( substring_index(recipients, ',', n), ',', -1 ) AS email FROM dashboards JOIN numbers ON char_length(recipients) - char_length(replace(recipients, ',', '')) >= n - 1 ) email_recipients_by_dashboard GROUP BY 1
31.3.20. TS: Unknown collation: utf8mb4_unicode_520_ci mysql:unknown collation
- Error encountered when importing .sql in phpMyAdmin
#1273 - Unknown collation: 'utf8mb4_unicode_520_ci'- Replace collation and upload .sql again
sed -i 's/utf8mb4_unicode_520_ci/utf8mb4_unicode_ci/g' file.sql
31.4. MariaDB
31.4.1. Install
sudo apt install mariadb-server # service will start automatically, check status sudo systemctl status mariadb # MariaDB version mysql -V # mysql Ver 15.1 Distrib 10.1.38-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 # 15.1 is the command line client version. 10.1.38-MariaDB is the mysql server version the client was built with # In order to get the mysql server version, have to do it in console: mysql:config # improve security: prompt for root user password, remove anonymous user, restrict root user access to local machine and remove the test database # Y to all sudo mysql_secure_installation
31.4.2. Config loading
- Hyphen vs Underscore
- In most cases, they have the same effect in cofig files. But config variables returned from queries are always underscore
- not exist in Ubuntu install
- loads other files. symlink to
/etc/alternatives/my.cnfsymlink to/etc/mysql/mariadb.cnf/etc/mysql/mariadb.cnf- global defaults. This file
- In MySQL
- /etc/mysql/mysql.cnf
/etc/mysql/conf.d/*.cnfglobal options
- In MySQL
- mysql.cnf
[mysql]empty- mysqldump.cnf
[mysqldump]not empty
/etc/mysql/mariadb.conf.d/*.cnfMariaDB-only options
- 50-server.cnf
[mysqld]bind-address- 50-client.cnf
- 50-mysqld_safe.cnf
- 50-mysql-clients.cnf
- In MySQL
/etc/mysql/mysql.conf.d/*.cnf
- mysqld.cnf
[mysqld_safe][mysqld]- mysqld_safe_syslog.conf
[mysqld_safe]
- (no term)
- Custom config files should have only one dot and extension is
.cnf
- by default not created. User-specific options
cd /etc/mysqlgrep -r 'sql_mode' .
# Default options are read from the following files in the given order: mysqld --help --verbose | less # Usually /etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf # see which defaults (after changes) mysqld is using mysqld --print-defaults
This does not work with Ubuntu. Change 50-server.cnf instead.
[mysqld] #skip-networking # default is no skip-networking directive # if it's there, MariaDB will not work with any connection with TCP/IP bind-address = 127.0.0.1 # default. 127.0.0.1 is also localhost. No one can connect to server from other hosts or from the same host over TCP/IP on a different interface than the loopback (127.0.0.1). # Which means only local connection and the host should be referred to 127.0.0.1 or localhost. # remove bind-address the directive by commenting out so that it binds to any IP `0.0.0.0` or `::` # it becomes with commeting out! #skip-networking #bind-address = 127.0.0.1 # or in the last conf file ~/.my.cnf or last in /etc/my.cnf [mysqld] skip-networking=0 skip-bind-address
31.4.3. Restart
systemctl enable mariadb systemctl status mariadb systemctl start mariadb systemctl stop mariadb
31.4.4. Manage users
-- see any users that can connect remotely SELECT User, Host FROM mysql.user WHERE Host <> 'localhost';
31.4.5. CREATE OR REPLACE [PROCEDURE|FUNCTION|TRIGGER]
31.4.6. Trigger
- List triggers in all dbs
SHOW TRIGGERS;- List triggers for a db
SHOW TRIGGERS FROM mydbname;- List triggers for a table
SHOW TRIGGERS FROM mydbname LIKE 'mytblname';orSHOW TRIGGERS FROM mydbname WHERE 'Table' = 'mytblname';
- See a specific trigger
SHOW TRIGGERS FROM mydbname WHERE `Trigger` = 'my_trigger_name'\G- Drop a trigger
DROP TRIGGER mydbname.mytriggername;- (no term)
- Can call stored procedures but cannot pass
NEWnorOLDas the whole row- See 31.3.11.1
DROP TRIGGER IF EXISTS [schema_name.]trigger_name DELIMITER // CREATE OR REPLACE TRIGGER trigger_newguid BEFORE UPDATE ON sp FOR EACH ROW BEGIN IF NEW.modifiedon <> OLD.modifiedon THEN -- Don't use `UPDATE trigger_newguid SET` because we are in a row -- `BEFORE UPDATE` is required otherwise the record/row is locked SET NEW.newguid = UuidToBin(uuid()); end if; END; // DELIMITER ; -- Update field modifiedon to trigger trigger_newguid -- newguid field will be updated before the modifiedon field is updated UPDATE sp SET modifiedon = NOW();
31.4.7. Stored function mariadb:stored function
- Refer to mariadb:uuid
- It can't be run in
CREATE TABLEALTER TABLE
31.4.8. UUID mariadb:uuid
- https://mariadb.com/kb/en/library/guiduuid-performance/
- MariaDB 10.4 and above may support MySQL 8.0 where extra UUID functions are defined
- Run this in console
DELIMITER // CREATE FUNCTION UuidToBin(_uuid BINARY(36)) RETURNS BINARY(16) LANGUAGE SQL DETERMINISTIC CONTAINS SQL SQL SECURITY INVOKER RETURN UNHEX(CONCAT( SUBSTR(_uuid, 15, 4), SUBSTR(_uuid, 10, 4), SUBSTR(_uuid, 1, 8), SUBSTR(_uuid, 20, 4), SUBSTR(_uuid, 25) )); CREATE FUNCTION UuidFromBin(_bin BINARY(16)) RETURNS BINARY(36) LANGUAGE SQL DETERMINISTIC CONTAINS SQL SQL SECURITY INVOKER RETURN LCASE(CONCAT_WS('-', HEX(SUBSTR(_bin, 5, 4)), HEX(SUBSTR(_bin, 3, 2)), HEX(SUBSTR(_bin, 1, 2)), HEX(SUBSTR(_bin, 9, 2)), HEX(SUBSTR(_bin, 11)) )); // DELIMITER ;
Then you can
CREATE TABLE t ( id binary(16) PRIMARY KEY, id_text varchar(36) GENERATED ALWAYS AS ( LCASE(CONCAT_WS('-', HEX(SUBSTR(id, 5, 4)), HEX(SUBSTR(id, 3, 2)), HEX(SUBSTR(id, 1, 2)), HEX(SUBSTR(id, 9, 2)), HEX(SUBSTR(id, 11)) )) ) virtual ); -- Letting MySQL create the UUID: INSERT INTO t (uuid, ...) VALUES (UuidToBin(UUID()), ...); -- Creating the UUID elsewhere: INSERT INTO t (uuid, ...) VALUES (UuidToBin(?), ...); -- Retrieving (point query using uuid): SELECT ... FROM t WHERE uuid = UuidToBin(?); -- Use the stored function on the righ hand side in WHERE clause. Don't do this -- WHERE UuidFromBin(uuid) = '1026-baba-6ccd780c-9564-0040f4311e29' -- Retrieving (other): SELECT UuidFromBin(uuid), ... FROM t ...;
31.5. MSSQL
31.5.1. Modify column values if exist (CASE WHEN)
SELECT colOne, varColTwo = CASE colTwo WHEN 1 THEN 1 WHEN 0 THEN 3 END, varImage = CASE WHEN len(colImage) > 0 THEN 'http://abc.com/images/' + colImage ELSE '' END FROM aTable ORDER BY varColTwo DESC, CASE WHEN colFour = 1 THEN colFive WHEN colFour = 2 THEN colSix END ASC
31.5.2. Concatenate Multiple Columns to Single Column as String
A doc has multiple categories. Concatenate the categories delimited by commas.
SELECT doc.intDocID ,doc.varTitle ,STUFF( (SELECT ',' + CAST( t2.catID AS varchar(10) ) FROM tblDocCat t2 WHERE t2.intDocID = doc.intDocID FOR XML PATH('') ), 1, 1, '' ) AS docCats
31.5.2.1. STUFF
This deletes the first letter of a column and returns the rest. STUFF first deletes a substring from aCol at start:1 and length_to_delete:1 and then insert a string at the start position
STUFF(aCol, 1,1, '')
31.5.3. Export as CSV - BCP mssql:bcp
- bcp.exe is in
C:\Program Files\Microsoft SQL Server\90\Tools\Binn\ Options
bcp "Select * from fullDatabasename.dbo.tableName" queryout "c:\aFolder\aFile.csv" -c -t"\",\"" -r"\"\n\"" -T
- c
- Ascii
- t
- field terminator. Comma
- r
- row terminator. New line
"\n" - T
- trusted connection
- DEDP225.[dev.todaystrucking].dbo.[tblDocument]
31.5.3.1. Escape % if bcp is in batch file
bcp "SELECT * FROM aTable WHERE email LIKE '%%@%%.%%'"
31.5.3.2. Save long query in Global Temporary table (GTT)
SELECT * INTO ##myglobaltemptable FROM tableA
Drop the GTT after bcp: DROP TABLE ##myglobaltemptable
bcp ##myglobaltemptable out "c:\folder\file.csv" -c -t"\",\"" -r"\"\n\"" -T
31.5.3.3. Get all columns
DECLARE @colnames VARCHAR(max); SELECT @colnames = COALESCE(@colnames + ',', '') + char(39)+ column_name +char(39) FROM INFORMATION_SCHEMA.COLUMNS where TABLE_NAME='%yourtablename%'; SELECT @colnames
It returns 'col1','col2',...,'colLast'
Export to header.csv
bcp "SELECT 'col1','col2',...,'colLast'" queryout "c:\header.csv" -c -t"\",\"" -r"\"\n\"" -T
31.5.3.4. Last step
Both header and data files need to be fixed. Insert a double quote at the beginning of the file and delete the last double quote at the end of it. Insert a new line at the end of the header file.
- Combine 2 files
copy /b c:\header.csv+c:\folder\file.csv c:\export.csv
31.5.4. Export as XML
- By default, it's UTF-8 encoded. datetime will be automatically converted to XML format
- If value is NULL, the xml node will not be generated
- By default, MSSQL will add
<CR><LF>into the Unicode XML data stream every 2033 characters - Use mssql:bcp with option -r to add no new line
31.5.4.1. Query
SELECT CustomerID as "@id", CompanyName, Address as "address/street", City as "address/city", Region as "address/region", PostalCode as "address/zip", Country as "address/country", ContactName as "contact/name", ContactTitle as "contact/title", NULLIF(Phone,'') as "contact/phone", -- If value is NULL, the xml node will not be generated Fax as "contact/fax" FROM Customers FOR XML PATH('Customer'), ROOT('doc')
Result
<doc> <Customer id="ALFKI"> <CompanyName>Alfreds Futterkiste</CompanyName> <address> <street>Obere Str. 57</street> <city>Berlin</city> <zip>12209</zip> <country>Germany</country> </address> <contact> <name>Maria Anders</name> <title>Sales Representative</title> <phone>030-0074321</phone> <fax>030-0076545</fax> </contact> </Customer> ... </doc>
INNER JOIN Multiple records
SELECT d.docID ,( SELECT c.category AS category FROM tblDocCats dc INNER JOIN tblCategory c ON c.catID = dc.catID WHERE dc.docID = d.docID FOR XML PATH(''), TYPE ) AS categories FROM doc d FOR XML PATH('item'), ROOT('items')
Result
<items> <item> <docID></docID> <categories> <category></category> <!--...--> </categories> </item> </items>
This
SELECT d.docID ,( SELECT c.category AS name, c.catID as ID FROM tblDocCats dc INNER JOIN tblCategory c ON c.catID = dc.catID WHERE dc.docID = d.docID FOR XML PATH('category'), TYPE ) AS category FROM doc d FOR XML PATH('item'), ROOT('items')
will result as
<items> <item> <docID></docID> <category> <name></name> <id></id> </category> <category> <name></name> <id></id> </category> ... </item> </items>
31.5.4.2. Different modes
FOR XML AUTO
<servername.dbo.Customers ID="C11" Name="..." Address="..." />
FOR XML RAW
<row ID="C11" Name="..." Address="..." />
FOR XML PATH
<row> <ID></ID> <Name></Name> <Address></Address> </row>
31.5.4.3. bcp
bcp "query..." queryout C:\file.xml -T -w -r
| r | no value means don't add any newline character for every 2033 characters or new row |
| w | Use Unicode characters |
| S | server_name[\instance_name] |
Save long query as stored procedure. You can't save FOR XML in Global Temporary Table. dev.todaystrucking is the database name. Inside the real SELECT, you can refer to table as dbo.tablename not DEDP225.[dev.todaystrucking].dbo.tablename
USE [dev.todaystrucking]
GO
IF OBJECT_ID('dbo.usp_exporttr') IS NOT NULL DROP PROC dbo.usp_exporttr
GO
CREATE PROC dbo.usp_exporttr AS
SET NOCOUNT ON
SELECT ...
FOR XML PATH('item'), ROOT('items')
RETURN
GO
Then creat another query in MS SQL Server Management Studio and run
EXEC xp_cmdshell 'bcp "EXEC [dev.todaystrucking].dbo.usp_exporttr" queryout "c:\a.xml" -S (local) -T -w -r'
You may receive "SQL Server blocked access to procedure 'sys.xp_cmdshell' of component 'xp_cmdshell' because this component is turned off as part of the security configuration"
Just go to bcp folder and run as a command line mssql:bcp
bcp "EXEC [dev.todaystrucking].dbo.usp_exporttr" queryout "c:\a.xml" -S (local) -T -w -r
The exported xml file will have UCS-2 encoding which is a preceding version of UTF-16. This encoding is in BOM. For SQL Server lower than 2016, you can't set encoding to UTF-8.
31.5.4.4. bcp clean up
You can run these in Git Bash on Windows Change encoding to UTF-8 iconv -f UCS-2 -t UTF-8 a.xml > b.xml
Add xml declaration echo '<?xml version="1.0"?>' | cat - b.xml > c.xml
Delete xml:invalid_characters sed sed "s///gi; s///gi;" c.xml > d.xml
31.5.5. Copy to Excel
C-a and C-c, C-v in Excel
For datetime, you can =now() in a new cell, format paint the datetime columns.
In the =now() cell, you can check out the Custom format. It's yyyy-mm-dd h:mm
You can change it to yyyy-mm-dd h:mm:ss AM/PM
31.5.6. Migrate to MySQL
Export MSSQL to a bak file. mssql:backup https://stackoverflow.com/questions/156279/how-to-import-a-sql-server-bak-file-into-mysql
If the bak file is less than 10gb, follow this: https://www.silicongadget.com/software/database/import-mssql-bak-files-to-mysql/2955/
- Restore MSSQL database using the bak file using MS SQL Server Express Edition
- Use MySQL Workbench to connect to the MSSQL database and then migrate to MySQL
31.5.7. Connections
Number of active connections by database
SELECT
DB_NAME(dbid) as DBName,
COUNT(dbid) as NumberOfConnections,
loginame as LoginName
FROM sys.sysprocesses
WHERE dbid > 0
GROUP BY dbid, loginame
-- 2017 new query
SELECT @@ServerName AS server
,NAME AS dbname
,COUNT(STATUS) AS number_of_connections
FROM sys.databases sd
LEFT JOIN sys.sysprocesses sp ON sd.database_id = sp.dbid
GROUP BY NAME
31.5.8. Last access
SELECT d.name, last_user_seek = MAX(last_user_seek), last_user_scan = MAX(last_user_scan), last_user_lookup = MAX(last_user_lookup), last_user_update = MAX(last_user_update) FROM sys.dm_db_index_usage_stats AS i JOIN sys.databases AS d ON i.database_id=d.database_id GROUP BY d.name
31.5.9. Backup
mssql:backup https://docs.microsoft.com/en-us/sql/relational-databases/backup-restore/create-a-full-database-backup-sql-server Export database to bak file. SQL Management Studio Tool > right click on a database and Tasks > Backup > Use Full backup, specify a destination (path to bak). Use overwrite settings in Options. Then click on Script > Script to file. You will get a .sql:
BACKUP DATABASE [aDatabase] TO DISK ='N'C:\afile.bak' WITH NOFORMAT, INIT, NAME = N'aDatabase-Full Database Backup', SKIP, NOREWIND, NOUNLOAD, STATS = 10 GO
Make filename dynamic (remove spaces if necessary):
DECLARE @filename nvarchar(200) = '' SELECT @filename = 'C:\test-' + convert(varchar(23), getdate(), 126) + '.bak' BACKUP DATABASE [aDatabase] TO DISK = @filename WITH NOFORMAT, INIT, NAME = N'aDatabase-Full Database Backup', SKIP, NOREWIND, NOUNLOAD, STATS = 10 GO
Run this .sql file:
sqlcmd -S sqlservername -i C:\sqlFileName.sql
31.6. MongoDB
- Installation
- https://docs.mongodb.com/manual/installation/ Community Edition
- Windows
- Naming convention
- camelCase for DB name
31.7. GraphQL
- GitHub GraphQL API explorer
- https://developer.github.com/v4/explorer/
31.8. ClickHouse
- Refer to docker:clickhouse
- Use uppercase for functions
INSERT INTO () VALUES (), don't useVALUE
32. Google Ad Manager - DFP
32.1. Default code - Google Publisher Tag - gpt.js
- After 2019-06-12, it's
https://securepubads.g.doubleclick.net/tag/js/gpt.js- Before 2019-06-12, it's
https://www.googletagservices.com/tag/js/gpt.js
- Before 2019-06-12, it's
- https://developers.google.com/doubleclick-gpt/reference
<head> <script async='async' src='https://securepubads.g.doubleclick.net/tag/js/gpt.js'></script> <script> var googletag = googletag || {}; googletag.cmd = googletag.cmd || []; </script> <script> googletag.cmd.push(function() { googletag.defineSlot('/123456/my_ad_unit_1', [300, 250], 'div-gpt-ad-1557260744066-0').addService(googletag.pubads()); googletag.pubads().enableSingleRequest(); googletag.enableServices(); }); </script> </head> <body> <!-- /123456/my_ad_unit_1 --> <div id='div-gpt-ad-1557260744066-0' style='height:250px; width:300px;'> <script> googletag.cmd.push(function() { googletag.display('div-gpt-ad-1557260744066-0'); }); </script> </div> </body>
32.1.1. Legacy
<head> <script> var googletag = googletag || {}; googletag.cmd = googletag.cmd || []; (function () { var gads = document.createElement('script'); gads.async = true; gads.type = 'text/javascript'; var useSSL = 'https:' == document.location.protocol; gads.src = (useSSL ? 'https:' : 'http:') + '//www.googletagservices.com/tag/js/gpt.js'; var node = document.getElementsByTagName('script')[0]; node.parentNode.insertBefore(gads, node); })(); googletag.cmd.push(function () { googletag.defineSlot('/123456/my_ad_unit_1', [300, 250], 'div-gpt-ad-unit-1') .addService(googletag.pubads()); googletag.pubads().collapseEmptyDivs(); googletag.pubads().enableSingleRequest(); googletag.enableServices(); }); </script> </head> <body> <!-- /123456/my_ad_unit_1 --> <div id='div-gpt-ad-unit-1' style='height:250px; width:300px;'> <script> googletag.cmd.push(function() { googletag.display('div-gpt-ad-unit-1'); }); </script> </div> </body>
32.2. Why async is required
- Video companion requires async
pubService.refreshrequires async- POST request requires async. GET request with larger than 2048 bytes not supported
- However, it's better to use sync mode for rich media creative unless the creative is in friendly frames
32.3. Competition Calculation
- List of candidate line items
- select the best line item
- select the best creative associated with the winning line item
- DFP remnant and Ad Exchange/Adsense line items are evaluated
to see if they have line items with a higher yield
- the best creastive associated with the winning line item is selected
- creative returned to the browser. Dynamic allocation.
32.4. Targeting - key value, Geo
32.4.1. Key-value
- Filter format
AND(site=MySiteA,NOT(exclusive=exclusive))
Slot-level targeting is recommended
googletag.defineSlot('/123/travel/asia/food', [728, 90], "div-id") .addService(googletag.pubads()) .setTargeting("interests", ["sports", "music"]);
Page-level targeting
googltag.pubads().setTargeting("topic", "basketball");
32.4.2. Geolocation
- Postcal code
- target Canada to the first 3 characters which is Forward Sortation Area (the last 3 are called Local Delivery Unit). The first character can show which province. Big provinces like ON and QC have multiple first characters
- City, province, country
- Some big city has multiple cities grouped together e.g. Vaughan city area has Concord, Maple, Thornhill and Woodbridge cities
32.5. Exclusive Ads on Specific Page
Let's say there're 2 LBs and 2 BBs throughout the website. 999/LB1, 999/LB2, 999/BB1, 999/BB2 For a specific page, Line Item A takes over all ad units. For another specific page, Line Item B takes over all ad units.
The perfect way Create multilevel ad units and make them Special Ad Units. 999/LB1/exclusive, 999/LB2/exclusive 999/BB1/exclusive, 999/BB2/exclusive
On those specific pages:
- Use the new tags of multilevel and special ad units on HTML instead of the original ones.
- setTargetting on HTML and on DFP so that Line Item A and B can be distinguished.
The better way Multilevel and Special ad units are only avaiable in Premium. Here's the way for SBS
- 999/LB1_exclusive, 999/LB2_exclusive
- 999/BB1_exclusive, 999/BB2_exclusive
- Use the new tags on HTML instead of the original ones
- setTargetting on HTML and DFP so that Line Item A and B
The workaround way If you don't have control on HTML, you can:
- For Line Item A and B
- On DFP
- add Key-Value pair exclusive is exclusive
- add Key-Value pair sponsor is lineitemA
- In HTML, for those specific pages
- setTargeting("exclusive","exclusive");
- setTargeting("sponsor","lineitemA");
- On DFP
- For other normal line items
- On DFP
- add Key-Value par 'exclusive is not
exclusive' (means contain and * means begin with)
- add Key-Value par 'exclusive is not
- In HTML, do nothing (remain the same)
- On DFP
32.6. Email - Non-JavaScript
- Tagless Request
- https://admanager.google.com/95740733#delivery/line_item/detail/order_id=2556961137&line_item_id=5091153061
Was called
Simply URL<!-- Email HTML --> <a href="http://my.com/ad/ad_unit_name/728x90/"> <img width="728" height="90" alt="" src="https://securepubads.g.doubleclick.net/gampad/ad?iu=/networkid/ad_unit_name&sz=728x90&c=[todays_date][account]" /> </a> <!-- Website --> <script src="//code.jquery.com/jquery-1.10.2.js"></script> <script> var ad_unit = 'ad_unit_name'; var ad_unit_size = '728x90'; var rnd=Math.floor(Math.random() * (999999 - 10 +1)) + 10; var theUrl='//pubads.g.doubleclick.net/gampad/adx?iu=/' + ad_unit + '&sz=' + ad_unit_size + '&c=' + rnd; var xmlHttp = null; xmlHttp = new XMLHttpRequest(); xmlHttp.open( "GET", theUrl ); xmlHttp.send(); xmlHttp.onreadystatechange=function() { if (xmlHttp.readyState==4 && xmlHttp.status==200) { xmlDoc = $.parseHTML( xmlHttp.response ); adlink = $(xmlDoc).find('a').attr('href'); if (typeof adlink == "undefined") adlink = "http://www.example.com"; window.location = adlink; } } </script>
PHP, DFP creative type Custom. Get response line by line
$url = "http://pubads.g.doubleclick.net/gampad/adx?iu=/12343/ad_unit_name&sz=230x20&c=". REQUEST_TIME."&m=text/html"; $json = @file_get_contents($url); if ($json !== FALSE) { $lines = explode("\n", $json); $link = ( isset($lines[0]) ) ? trim($lines[0]) : 'default'; }
WordPress example: wp:filter:template_include GitHub Gist: https://gist.github.com/levonlee/888f5f5b0a33acebe2b20ecbbff3b106
- Base URL
https://securepubads.g.doubleclick.net/gampad/[request-type]?[parameters]- Before
https://pubads.g.doubleclick.net/gampad/[request-type]?[parameters]orhttp://- (no term)
- Request Type
- ad
- simple image creative
- adx
- return raw code of creative which is to be placed inside an iframe with JS available
- jump
- log click and redirect to destination URL
- clk
- only log click but no redirect. Only in DFP Premium
- (no term)
Parameters
iu for all request types sz for ad, jump and adx t for ad, jump and adx. key-value pairs excl_cat competitive exclusions c for ad, jump and adx. cache buster. only number m for adx. Specify file type to return. m=text/html submodel for all request types. Mobile device name u_w for all request types. Mobile device screen height. title required if mutliple ad tags use the same ad unit code on the same page mob Indicaste it's a mobile request.
- (no term)
- Delay impression counting. Choose one
Add URL parameters
&d_imp=1&d_imp_hdr=1in the requests (adoradx)d_imp- if enabled, impression counting upon request is disabled
d_imp_hrd- if enabled, the http response header contains the url needed to ping
Google-Delayed-Impressionresponse header- Ad Manager impression URL
Google-3rdParty-Delayed-Impressionresponse header- 3rd party impression URL, if present in creative
- Ping the URL with http request header
Google-Delayed-Impression - an "Ad server impression" and "Ad server downloaded impression" is recorded
- Note
adrequest does not return anything. Useadxrequest
- Ping the URL with http request header
32.7. Responsive
- Add a size to the ad unit /networkid/adunitname
- Specify one or more sizes for a line item that uses this ad unit
- Upload each creative and specify the size. You can have creative bigger than the size you specify
googletag.cmd.push(function() { var leaderboard_mapping = googletag.sizeMapping() .addSize([1280, 800], [[970, 250], [970, 90], [728, 90]]) .addSize([600, 800], [468, 60]) // viewport size is width and height both > 600 and 800 .addSize([0, 0], [[320, 50], [320, 100]]) // Map any viewport size /* This will only display the ad if it's mobile or tablet viewport size .addSize([0, 0], []) // [] means don't try to call an ad .addSize([320, 700], [300, 250]) // Mobile or tablet .addSize([1050, 200], []) */ .build(); googletag.defineSlot('/networkid/adunitname' , [728, 90] // If there is an error in the mapping or if the browser size can't be determined // the size specified here in .defineSlot will be used. , 'div-id') .defineSizeMapping(leaderboard_mapping) .addService(googletag.pubads()); });
32.8. Macro in Third Party Code, Custom Code
CACHEBUSTER- time in integer
CLICK_URL_UNESC- normal click macro e.g.
%%CLICK_URL_UNESC%%https://adserver.ca/destination%%CLICK_URL_UNESC%%%%DEST_URL%%
CLICK_URL_ESC- escaped click macro (&, ? , %). The macro is passed as a parameter to 3rd party server
https://adserver.ca/destination?ncu=%%CLICK_URL_ESC%%
VIEW_URL_UNESC, VIEW_URL_ESC- required in Custom Code
%%VIEW_URL_UNESC%%%%FILE:JPG1%%
- (no term)
Example dfp:third party code Third Party Code with macro automatically inserted
<!-- url parameter ncu is inserted, ord's [timestamp] is replaced by CACHEBUSTER --> <script src="http://bs.serving-sys.com/Serving/adServer.bs?c=28&cn=display&pli=1074030166&w=728&h=90&ncu=$$%%CLICK_URL_UNESC%%$$&ord=%%CACHEBUSTER%%&ifrm=-1&z=0"></script> <noscript> <!-- insert CLICK_URL_UNESC in href --> <a href="%%CLICK_URL_UNESC%%http://bs.serving-sys.com/Serving/adServer.bs?cn=brd&pli=1074030166&Page=&Pos=944448101" target="_blank"> <img src="http://bs.serving-sys.com/Serving/adServer.bs?c=8&cn=display&pli=1074030166&Page=&Pos=944448101" border=0 width=728 height=90></a> </noscript>
VIEW_URL_UNESCorVIEW_URL_ESCshould be included otherwise impression won't be tracked.- In preview, this view macro returns empty but that's normal. It has value when the creative is served.
- Sometimes you need to have DFP inserted the macros in third party code mode, and then use that in Custom Code.
32.9. Creative types
32.9.1. Creative Type: Third Party Code - Video
- Video is hosted on third party server
- Video player size often doesn't match the ad unit size e.g. big box
Creative examples on DFP. Refer to video:jw
<!-- JW script tag --> <script src="//content.jwplatform.com/players/MEDIAID-PLAYERID.js?v=%%CACHEBUSTER%%"></script> <!-- JW iframe tag --> <div style="position:relative; padding-bottom:56.25%; overflow:hidden;"><iframe src="https://cdn.jwplayer.com/players/MEDIAID-PLAYERID.html?v=%%CACHEBUSTER%%" width="100%" height="100%" frameborder="0" scrolling="auto" allowfullscreen style="position:absolute;"></iframe></div> <!-- Viewbix --> <iframe width="300" height="250" src="http://www.viewbix.com/frame/1234-567-890?w=300&h=250&ord=%%CACHEBUSTER%%" frameborder="0" scrolling="no" allowTransparency="true"></iframe>
32.9.2. Creative Type: HTML5
Upload zip file or single html file
32.9.3. Creative Type: DoubleClick Tag, DCM Tag
DCM, previously known as DFA (for advertisers), is DoubleClick Campaign Manager, part of the DoubleClick Marketing Solutions.
Always use Internal redirect tag for DFP. Sometimes this tag is not given by agency, you will need to convert it to this type. Ins tag is recommended for ad servers other than DFP because it has active view capability.
ddm/jump tag (redirect to final destination) has to be loaded with ddm/ad tag.
Other tags like ddm/trackimp, ddm/trackclk, ddm/clk don't have to be loaded with ddm/ad tag.
32.9.3.1. Standard tags (ddm/jump, ddm/ad)
<A HREF="https://ad.doubleclick.net/ddm/jump/Nxxxx.site-keyname/Byyyyyyy.Pzzzz;sz=widthxheight;kw=[keyword];ord=[timestamp]?"> <IMG SRC="https://ad.doubleclick.net/ddm/ad/Nxxxx.site-keyname/Byyyyyyy.Pzzzz;sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I?" BORDER=0 WIDTH=X HEIGHT=Y ALT="Click Here"></A>
Nxxx: DCM account id Byyyy: DCM campaign ID .Pzzzz: DCM placement ID dc_lat=N: key-value pair for mobile app to pass if the user has enabled "Limit Ad Tracking" dc_rdid=Czzz: key-value pair for mobile app to pass resettable device identifiers in the form of IDFA for iOS or advertising ID (AdID) for Android tag_for_child_directed_treatment=I: key-value pair for mobile app to pass info about if a request may come from a user under the age of 13
32.9.3.2. iframe/javascript tags (ddm/adi, ddm/adj)
<IFRAME SRC=""https://ad.doubleclick.net/ddm/adi/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"" WIDTH=300 HEIGHT=250 MARGINWIDTH=0 MARGINHEIGHT=0 HSPACE=0 VSPACE=0 FRAMEBORDER=0 SCROLLING=no BORDERCOLOR='#000000'> <SCRIPT language='JavaScript1.1' SRC=""https://ad.doubleclick.net/ddm/adj/N1234.123456yoursite.com/B1234567.123456789;abr=!ie;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?""> </SCRIPT> <NOSCRIPT> <A HREF=""https://ad.doubleclick.net/ddm/jump/N1234.123456yoursite.com/B1234567.123456789;abr=!ie4;abr=!ie5;sz=300x250;ord=[timestamp]?""> <IMG SRC=""https://ad.doubleclick.net/ddm/ad/N1234.123456yoursite.com/B1234567.123456789/B10762433.143897886;abr=!ie4;abr=!ie5;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"" BORDER=0 WIDTH=300 HEIGHT=250 ALT=""Advertisement""></A> </NOSCRIPT> </IFRAME>
32.9.3.3. javascript (ddm/adj)
<SCRIPT language='JavaScript1.1' SRC=""https://ad.doubleclick.net/ddm/adj/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?""> </SCRIPT> <NOSCRIPT> <A HREF=""https://ad.doubleclick.net/ddm/jump/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp]?""> <IMG SRC=""https://ad.doubleclick.net/ddm/ad/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"" BORDER=0 WIDTH=300 HEIGHT=250 ALT=""Advertisement""></A> </NOSCRIPT>
32.9.3.4. Pre-fetch tags (ddm/pfadx)
# VAST 2.0 https://ad.doubleclick.net/ddm/pfadx/Nxxxx.site-keyname/Byyyyyyy;kw=[keyword]; sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I;dcmt=text/xml # VAST 3.0 https://ad.doubleclick.net/ddm/pfadx/Nxxxx.site-keyname/Byyyyyyy;kw=[keyword]; sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I;dcmt=text/xml;dc_vast=3 # VAST 4.0 https://ad.doubleclick.net/ddm/pfadx/Nxxxx.site-keyname/Byyyyyyy;kw=[keyword]; sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I;dcmt=text/xml;dc_vast=4
32.9.3.5. Click tracker (ddm/clk)
https://ad.doubleclick.net/ddm/clk/[ad ID];[placement ID];[verifier]?[click-through URL]
32.9.3.6. Internal redirect tag
https://ad.doubleclick.net/ddm/ad/N1234.123456yoursite.com/B1234567.123456789;sz=320x548
# Image URL https://ad.doubleclick.net/ddm/ad/Nxxxx.site-keyname/Byyyyyyy.n;sz=widthxheight;dc_expa=URL # Click-through URL https://ad.doubleclick.net/ddm/jump/Nxxxx.site-keyname/Byyyyyyy.n;sz=widthxheight
dc_expa: Ping an encoded URL in order to track real-time expansions of rich media display expanding creatives.
Grab the attribute data-dcm-placement value from either iframe or Ins tag and also the size
https://ad.doubleclick.net/ddm/ad/[data-dcm-placement];sz=widthxheight
32.9.3.7. Ins tag
In DFP, always use internal redirect tag. May also deploy DCM tag as a custom creative by adding DFP macros. Instructions
Iframe
<ins class='dcmads' style='display:inline-block;width:300px;height:250px'
data-dcm-placement='N1234.123456yoursite.com/B1234567.123456789'
data-dcm-rendering-mode='script'
data-dcm-https-only
data-dcm-resettable-device-id=''
data-dcm-app-id=''>
<script src='https://www.googletagservices.com/dcm/dcmads.js'></script>
</ins>
Javascript
<ins class='dcmads' style='display:inline-block;width:300px;height:250px'
data-dcm-placement='N1234.123456yoursite.com/B1234567.123456789'
data-dcm-rendering-mode='script'
data-dcm-https-only
data-dcm-resettable-device-id=''
data-dcm-app-id=''>
<script src='https://www.googletagservices.com/dcm/dcmads.js'></script>
</ins>
32.10. Examples
32.10.1. Dynamic load new containers
googletag.cmd.push(function() { var n = 'div-id-my-slot-name'; // slot name var s = googletag.defineSlot('/95740733/float_comp_1', [300, 250], n).addService(googletag.pubads()); // slot googletag.display(n); // if previously in header, before enableServices, disableInitialLoad() is used // googletag.pubads().disableInitialLoad(); // googletag.enableServices(); // then follow first .display and refresh // googletag.pubads().refresh([s]); };
32.10.2. Wallpaper, Site Skin
On DFP
<script type="text/javascript"> (function() { 'use strict'; parent.jQuery(document).ready(function() { var backgroundAdwidth = 480; // TODO: find out the background image width var backgroundAdheight = 860; // TODO: find out the background image height var backgroundAdLeftImage = '%%VIEW_URL_UNESC%%%%FILE:JPG1%%'; var backgroundAdRightImage = '%%VIEW_URL_UNESC%%%%FILE:JPG2%%'; var backgroundAdLinkLeft = '%%CLICK_URL_UNESC%%%%DEST_URL%%'; var backgroundAdLinkRight = backgroundAdLinkLeft; var backgroundAdMainContentDivWidth = 996; var backgroundAdExtraTopSpace = 100; // Set to 0 if you don't want extra top space parent.jQuery('body').prepend('<div id="background-ad-right"></div><div id="background-ad-left"></div>'); var jsLink1 = "<script type='text/javascript'>" + "var backgroundAdwidth = " + backgroundAdwidth + ";" + "var backgroundAdheight = " + backgroundAdheight +";" + "var backgroundAdMainContentDivWidth = " + backgroundAdMainContentDivWidth + ";" + "var backgroundAdExtraTopSpace = " + backgroundAdExtraTopSpace + ";" + "</scr" + "ipt>"; parent.jQuery('head').append(jsLink1); var jsLink2 = parent.jQuery("<script type='text/javascript' src='http://yours.com/js/backgroundskinad.js'>"); parent.jQuery('head').append(jsLink2); var cssLink = parent.jQuery("<link rel='stylesheet' href='http://yours.com/css/media-w1200-w992.css'>"); parent.jQuery('head').append(cssLink); parent.jQuery('#background-ad-left, #background-ad-right').css({ 'position': 'fixed', 'z-index': '1', 'cursor': 'pointer', 'height': backgroundAdheight }); parent.jQuery('#background-ad-left').css({ 'background': 'url(' + backgroundAdLeftImage + ') no-repeat right top' }); parent.jQuery('#background-ad-right').css({ 'background': 'url(' + backgroundAdRightImage + ') no-repeat left top' }); parent.jQuery('#background-ad-left').click(function() { window.open(backgroundAdLinkLeft); }); parent.jQuery('#background-ad-right').click(function() { window.open(backgroundAdLinkRight); }); }); })(); </script>
On Web Server - backgroundskinad.js
jQuery(document).ready(function($) { var backgroundAdMainContentDivWidth = 996; var backgroundAdExtraTopSpace = 0; function setBackgroundAdPositionX() { var adPosition = new Array(); var viewportWidth = $(window).width(); var leftSpace = (viewportWidth-backgroundAdMainContentDivWidth)/2; if (leftSpace > 0) { if (leftSpace <= backgroundAdwidth) { adPosition['adWidthRealtime'] = leftSpace; adPosition['adHorizontalSpacing'] = 0; } else { adPosition['adWidthRealtime'] = backgroundAdwidth; adPosition['adHorizontalSpacing'] = leftSpace - backgroundAdwidth; } } else { adPosition['adWidthRealtime'] = 0; adPosition['adHorizontalSpacing'] = 0; } $('#background-ad-left').css({ 'left': adPosition['adHorizontalSpacing'], 'width': adPosition['adWidthRealtime'] }); $('#background-ad-right').css({ 'right': adPosition['adHorizontalSpacing'], 'width': adPosition['adWidthRealtime'] }); return adPosition; } function setBackgroundAdPositionY() { var topspace = backgroundAdExtraTopSpace - $(window).scrollTop(); // nav HTML element height if ($('#newcom-header-masthead').length && $('#newcom-header-masthead').height()) { topspace += $('#newcom-header-masthead').height(); } topspace = (topspace > 0) ? topspace : 0; $('#background-ad-left, #background-ad-right').css('top', topspace + 'px'); } function wallpaperFixViewport() { $('.cover-slogan, .cover-logo').addClass("m-clear"); $('#newcom-single-left').toggleClass("col-md-9 col-md-8"); $('#newcom-single-right').toggleClass("col-md-3 col-md-4"); $('#newcom-page-news-middle').toggleClass("col-md-7 col-md-6"); $('#newcom-page-news-right').toggleClass("col-md-3 col-md-4"); $('#newcom-taxonomy-videoseries-middle').toggleClass("col-md-7 col-md-6"); $('#newcom-content-sidebar-page-left').toggleClass("col-md-9 col-md-8"); $('#newcom-content-sidebar-page-right').toggleClass("col-md-3 col-md-4"); $('#newcom-page-middle').toggleClass("col-md-7 col-md-6"); $('#newcom-page-right').toggleClass("col-md-3 col-md-4"); $('#newcom-search-middle').toggleClass("col-md-7 col-md-6"); $('#newcom-search-right').toggleClass("col-md-3 col-md-4"); $('div.row div.col-md-8 div.row div.col-md-3 div.well').toggleClass("pa-sm"); $('#div-gpt-placeholder-1, #div-gpt-placeholder-2').toggleClass('center-block'); // Enlarge column of a parent if it has a child var videoDiv = $( "#yourspecialchild" ).parent( ".the-content" ).parent( ".col-md-9" ); if (videoDiv.length) { videoDiv.toggleClass("col-md-9 col-md-12"); // Move var titleDiv = videoDiv.prev(); if (titleDiv.length) { titleDiv.toggleClass("col-md-3 col-md-12"); titleDiv.insertAfter(videoDiv); } } } function setBackgroundMobile() { if (typeof backgroundAdMobile300x90 !== 'undefined' && backgroundAdMobile300x90) { var ad = jQuery('<a href="' + backgroundAdLink +'" target="_blank"></a>').append(jQuery('<img />',{ width: 300, height: 90, src: backgroundAdMobile300x90 })); if (jQuery("#background-ad-mobile").length) { jQuery("#background-ad-mobile").append(ad); } else { var subpages = '#wrapper div.row div.col-md-8'; if (jQuery(subpages).length) { var html = jQuery('<div id="background-ad-mobile"></div>').append(ad); jQuery(subpages).first().prepend(html); } } } } $(window).load(function() { wallpaperFixViewport(); // setBackgroundMobile(); setBackgroundAdPositionX(); setBackgroundAdPositionY(); $("#div-gpt-placeholder-0-oop").hide(); }); $(window).resize(function() {setBackgroundAdPositionX();}); $(window).scroll(function() {setBackgroundAdPositionY();}); });
media-w1200-w992.css
@media (min-width:1200px){ .col-lg-1, .col-lg-2, .col-lg-3, .col-lg-4, .col-lg-5, .col-lg-6, .col-lg-7, .col-lg-8, .col-lg-9, .col-lg-10, .col-lg-11, .col-lg-12{ width:auto; float:none; } .container{ width:990px } .col-md-1, .col-md-2, .col-md-3, .col-md-4, .col-md-5, .col-md-6, .col-md-7, .col-md-8, .col-md-9, .col-md-10, .col-md-11{ float:left } .col-md-12{ width:100% } .col-md-11{ width:91.66666666666666% } .col-md-10{ width:83.33333333333334% } .col-md-9{ width:75% } .col-md-8{ width:66.66666666666666% } .col-md-7{ width:58.333333333333336% } .col-md-6{ width:50% } .col-md-5{ width:41.66666666666667% } .col-md-4{ width:33.33333333333333% } .col-md-3{ width:25% } .col-md-2{ width:16.666666666666664% } .col-md-1{ width:8.333333333333332% } .col-md-pull-12{ right:100% } .col-md-pull-11{ right:91.66666666666666% } .col-md-pull-10{ right:83.33333333333334% } .col-md-pull-9{ right:75% } .col-md-pull-8{ right:66.66666666666666% } .col-md-pull-7{ right:58.333333333333336% } .col-md-pull-6{ right:50% } .col-md-pull-5{ right:41.66666666666667% } .col-md-pull-4{ right:33.33333333333333% } .col-md-pull-3{ right:25% } .col-md-pull-2{ right:16.666666666666664% } .col-md-pull-1{ right:8.333333333333332% } .col-md-push-12{ left:100% } .col-md-push-11{ left:91.66666666666666% } .col-md-push-10{ left:83.33333333333334% } .col-md-push-9{ left:75% } .col-md-push-8{ left:66.66666666666666% } .col-md-push-7{ left:58.333333333333336% } .col-md-push-6{ left:50% } .col-md-push-5{ left:41.66666666666667% } .col-md-push-4{ left:33.33333333333333% } .col-md-push-3{ left:25% } .col-md-push-2{ left:16.666666666666664% } .col-md-push-1{ left:8.333333333333332% } .col-md-offset-12{ margin-left:100% } .col-md-offset-11{ margin-left:91.66666666666666% } .col-md-offset-10{ margin-left:83.33333333333334% } .col-md-offset-9{ margin-left:75% } .col-md-offset-8{ margin-left:66.66666666666666% } .col-md-offset-7{ margin-left:58.333333333333336% } .col-md-offset-6{ margin-left:50% } .col-md-offset-5{ margin-left:41.66666666666667% } .col-md-offset-4{ margin-left:33.33333333333333% } .col-md-offset-3{ margin-left:25% } .col-md-offset-2{ margin-left:16.666666666666664% } .col-md-offset-1{ margin-left:8.333333333333332% } }
32.10.3. Interstitial Ad
- Interstitial as JW Player is a bit complicated
- Gist
- (no term)
- Load third party code into 1x1 floating ad unit. Insert DFP macro as a third party code first. Refer to dfp:third party code
- (no term)
On website
// Outside of the div var dfpSlots = {}; // non single request mode googletag.cmd.push(function() { // non single request mode dfpSlots.floating = googletag.defineOutOfPageSlot('/123456/floating', 'interstitial').addService(googletag.pubads()); // single request mode // googletag.defineOutOfPageSlot('/123456/floating', 'interstitial').addService(googletag.pubads()); googletag.pubads().addEventListener('slotRenderEnded', function (event) { if (event.slot.getAdUnitPath() === '/123456/floating') { document.getElementById('interstitial').style.display = 'none'; } }); // ... googletag.pubads().enableSingleRequest(); // Disable initial load, we will use refresh() to fetch ads. // Calling this function means that display() calls just // register the slot as ready, but do not fetch ads for it. // googletag.pubads().disableInitialLoad(); googletag.enableServices(); });
<div id="interstitial"> <script type="text/javascript"> googletag.cmd.push(function() { googletag.display("interstitial"); }); </script> </div>
- (no term)
On DFP, Custom Creative Type postMessage to website to create jQuery UI dialog and later receives a message to create content inside the iframe
<script type="text/javascript"> (function() { 'use strict'; var temp = '%%VIEW_URL_UNESC%%'; // Custom Code requires VIEW_URL_UNESC or VIEW_URL_ESC macro.. window.onload = function() { function receiveMessage(e) { if (e.data == 'load_interstitial_ad') { var html = '<div style="width:640px;height:480px;">' + '<script src="insert third party code"'+'>'+'</'+'script>'+ '<noscript>' + '<a href="insert_third_party_code" target="_blank"><img src="insert_third_party_code" border="0" width="640" height="480"></a>'+ '</noscript>'+ '</div>'; document.write(html); } } window.addEventListener('message', receiveMessage); var sendData = {action:"dfp_load_interstitial_ad", width:"640", height:"480"}; parent.postMessage(sendData,'*'); } })(); </script>
- (no term)
On website Once received message from iframe, load internal script and only load it once
var dfpInterstitial = {}; dfpInterstitial.loadCount=0; function newcomReceiveMessage(e) { if (e.data == 'dfp_load_interstitial_ad' && dfpInterstitial.loadCount == 0) { dfpInterstitial.loadCount++; dfpInterstitial.width =e.data.width; dfpInterstitial.height =e.data.height; // var cssLink = parent.jQuery("<link rel='stylesheet' href='https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/themes/ui-lightness/jquery-ui.css'>"); // jQuery('head').append(cssLink); ['//ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/themes/ui-darkness/jquery-ui.css', '/wp-content/themes/xxx/css/interstitial.css'].forEach(function (src) { var _css = document.createElement("link"); _css.rel = 'stylesheet'; _css.type = 'text/css'; _css.href = src; document.head.appendChild(_css); }); [ 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/jquery-ui.min.js','/sites/all/themes/abc/js/interstitial.js' ].forEach(function(src) { var script = document.createElement('script'); script.src = src; script.async = false; document.head.appendChild(script); }); // var jsLink = jQuery("<script type='text/javascript' src='http://yourwebsite.io/interstitial.js'>"); // jQuery('head').append(jsLink); } } window.addEventListener('message', newcomReceiveMessage, false);
- (no term)
On website, interstitial.js Make ad unit div as a dialog using jQuery UI. jqueryui:dialog jQuery UI Dialog runs script inside the container twice. The script is setup in DFP which only posts or receives message Even though the same message is posted twice but
dfpInterstitialCountensures this js only runs oncejQuery(function(){ var targetDialog = '#interstitial'; // TODO var iframeId = jQuery(targetDialog).find('iframe').get(0).id; var adWidth = dfpInterstitial.width; var adHeight = dfpInterstitial.height; jQuery(targetDialog).dialog({ width: 'auto', height: 'auto', resizable: false, draggable: false, closeOnEscape: true, modal: true, create: function (event, ui) { jQuery("#ui-dialog-title-dialog").hide(); jQuery(".ui-widget-content").css({"background-color":"transparent","background":"transparent","border":"none"}); jQuery(".ui-dialog-titlebar").css({"background-color":"transparent","background":"transparent","border":"none"}); jQuery(".ui-dialog-titlebar").removeClass('ui-widget-header'); // In Single Request Mode (initial load), every time .display() is called, the div will be refilled with new ad // In Non-Single-request mode (after initial load), need to do .refresh() if (typeof dfpSlots !== "undefined" && typeof dfpSlots.floating !== "undefined") { googletag.cmd.push(function() { googletag.pubads().refresh([dfpSlots.floating]); }); } jQuery(targetDialog).find('iframe').width(adWidth); jQuery(targetDialog).find('iframe').height(adHeight); }, open: function() { jQuery('.ui-widget-overlay').addClass('custom-overlay'); jQuery(this).closest(".ui-dialog") .find(".ui-dialog-titlebar-close") .removeClass("ui-dialog-titlebar-close") .html("<span class='ui-button-icon-primary ui-icon ui-icon-closethick'></span>"); jQuery(".ui-dialog-titlebar button").css({"border":"none"}); jQuery('.ui-widget-overlay').bind('click',function(){ jQuery(targetDialog).dialog('close'); }) }, close: function() { jQuery('.ui-widget-overlay').removeClass('custom-overlay'); } }); setTimeout(function () { jQuery(targetDialog).show(); var receiver = document.getElementById(iframeId).contentWindow; receiver.postMessage('load_interstitial_ad', '*'); }, 3000); });
.ui-widget-overlay.custom-overlay { background-color: black; background-image: none; opacity: 0.7; }
32.11. Pre-roll or Linear Video Ad
- The ad unit should have a Video (VAST) size e.g.
640x360v - In the line item, select the same video size as the inventory size and also select the ad unit
- Add a new Creative Set and select Linear then Video (not VPAID) for pre-roll, mid-roll or post-roll
- Upload the video file that a higher resolution e.g. 1920x1080. DFP will create all lower resolution videos
- Your video creative is setup. You need to generate the tag for the ad unit
- Use Video Suite Inspector to test the video ad. If the ad unit is newly created, it could take an hour to preview the video ad..
- Delivery > Troubleshoot > Video simulates the request taking under/over delivery into account
- DFP Video Tag
https://pubads.g.doubleclick.net/gampad/ads?sz=640x360&iu=/{network_id}/{VAST_code_name}&ciu_szs=640x90&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&url=[referrer_url]&description_url=[description_url]&correlator=[timestamp]- In the tag, use JW Player Targeting Macros to add values into these url parameters. Refer to video:jw
- The url of the page where the ad will appear
&url=__page-url__, and other macros&correlator=__timestamp__,&description_url=__referrer__
- The url of the page where the ad will appear
- Final DFP tag in JW is
https://pubads.g.doubleclick.net/gampad/ads?sz=640x360&iu=/{network_id}/{VAST_code_name}&ciu_szs=640x90&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&url=__page-url__&description_url=__referrer__&correlator=__timestamp__ In JW, specify the div id which the companion ad will be in and on the website
<div id="newcomjwplayerbanner" style="width:640px;height:90px;margin-bottom: 10px;"></div> <div id="newcomjwplayer"> <script type="application/javascript" src="//content.jwplatform.com/players/{video_id}-{player_id}.js"></script> </div>
32.12. Native Ad, Native Styles, Custom Rendering
Custom Rendering can only be used in mobile apps.
Create a native ad format under Delivery > Creatives > Native Styles Associate this native format with an ad unit. A native format is basically a HTML/CSS template with some placeholders which can be later defined when you upload a creative of Native type to a line item. Create a line item with Inventory sizes = Standard and Native format, and target the ad unit. Upload a creative of Native type. Specify the placeholders.
32.13. SafeFrame API
- Preview creative doesn't enable SafeFrame. The creative has to be deployed
- When console shows FriendlyFrame, it shows the container is not SafeFrame
- It's available for 4 types of creative
- custom
- d:on
- third-party
- d:on
- system-defined and user-defined templates
- d:off
- http://publisherconsole.appspot.com/safeframe/creative-preview.html
SafeFrame methods Inside Custom creative
<div id="container"> <script> function updateInViewPercentage() { var text = $sf.ext.inViewPercentage() + '%'; document.getElementById('percentage').innerHTML = text; } $sf.ext.register(728, 90, function(status, data) { // 728,90 are inital size of the creative // register is needed for calling other $sf.ext.xxx // Do nothing. This test doesn't make use of this callback. }); </script> <span> <strong>$sf.ext.inViewPercentage</strong> </span> <button onclick="updateInViewPercentage();">In view pecentage</button> <div id="percentage"></div> </div>
32.14. googletag.Service
- Methods
addEventListener(eventType, listener)- refer to google:gam:gpt:event
getSlots()
googletag.pubads()is agoogletag.PubAdsServicewhich extendsgoogletag.ServicegetTargetingKeys()- e.g.
['publisher', 'site', 'section', 'exclusive', 'sponsor'] getTargeting(key)- return array e.g.
getTargeting('exclusive')
32.15. googletag.Slot
- Methods
addService(service)getAdUnitPath()getSlotElementId()setCollapseEmptyDiv(collapse, opt_collapseBeforeAdFetch)- targeting set on Service level is not assigned here. Refer to google:gam:gpt:service
- same as above
32.16. Events
googletag.events.Eventinterface- https://developers.google.com/doubleclick-gpt/reference#googletag.events.Event
- serviceName
- slot
- google:gam:gpt:slot
- (no term)
googletag.evetns.Eventextension for certain eventsgoogletag.events.SlotRenderEndedEvent- fired when the creative code is injected into a slot. Before the creative's resources are fetched. Extra fields:
isEmpty- true if no ad was returned for the slot, false otherwise
googletag.pubads().addEventListener('slotRenderEnded', function (event) { if (event.slot.getAdUnitPath() === '/123456/adunit_name') { document.getElementById('div-gpt-ad-1403025754774-0-oop').style.display = 'none'; } });
32.17. Google Publisher Console
- To open, append to URL
?googfc=1(display console after page is loaded) or?google_console(keyboard to toggle console display)C-F10 - Or make a bookmark
javascript: googletag.openConsole();
32.18. Passback tag
Situation 1
- The webpage makes a call to the DFP ad server using the DFP ad tag
- DFP ad server returns an ad containing a third-party ad tag
- Third-party ad tag calls the third-party ad server for an ad
- Third-party ad server doesn't have an eligible ad, so returns a passback ad tag
- Passback ad tag makes a call to DFP to serve an ad matching the specified targeting criteria
- DFP server returns an ad that matches the passback ad tag targeting criteria
Situation 2
- Passback ad tag makes a call to DFP to serve an ad matching the specified targeting criteria
- DFP server returns an ad that matches the passback ad tag targeting criteria
32.19. Anti AdBlocker
32.20. Custom Fields
- Custom fields can be used in google:gam:query:filter and as dimensions in reports
- As query filter can only add filter to one field e.g. only filter 1 pair of key-value. Instead, create multiple Custom Fields for query filter
- can't be changed after set. only one
- Order
- Line Item
- Creative
- Editable, Read-only, Hidden (API can submit values)
- can't be changed after set. 30 custom fields to each object type
- Number
- select from either Yes or No
- enter any text value
- from a set of options that you configure. You must first create and save custom fields of the drop-down type, then navigate back to the custom field in order to configure these options
32.21. Reports
- Query filter is to filter the request google:gam:query:filter
- Dimensions and Metrics are also related to the request
For example, a request might have 2 key-value targeting. If you have this query
- Filter
- none
- Dimension
- key-value and line item
- Metric
- impression and click
The request will end up in 2 places
| key-value | line item | Impression | Click |
|---|---|---|---|
| a=xyz | LI_1 | 123 | 5 |
| b=ijk | LI_1 | 123 | 5 |
Add filter key-value contains a=xyz
- Filter out b=ijk
32.21.1. Downloaded impression
- Active View eligible impressions
- if creative has Active View enabled and the impression is counted with a downloaded pingback
- Active View measurable impressions
- close to 100% of above. A fail factor may be cross-domain iframe rendering
- Active View viewable impressions
- >= 50% area is displayed for >= 1 second. In-stream video: 50% for 2 seconds
- Ad server impressions
- counted after the ad is downloaded in the user's device. Doesn't require the ad content be fully loaded. Exclude from Ad Exchange and AdSense. Takes 30 mins for new ones to be recorded and displayed
- Ad server downloaded impressions
- Discontinued. Counted after the ad has started to load on the website. Doesn't require the ad content be fully loaded
- (no term)
- DFP provides a
System QuerycalledDownloaded Impressionsto compare with the old impression counting - (no term)
- Also now a new metric called
Request Typeto divide:- Goolge Publisher Tag
- Video Tag
- GPT Simple URL
- GPT Light
- Amp Ad Tag
32.22. Check available inventory - Forecasting
- Works only for Priority:Standard and Priority:Sponsorship (guaranteed line items) line items
- Priority:Sponsorship line items that are Paused, Draft or with no creatives are also in the competition
- Delivery > Orders or Delivery > Line items. Do one of the following
- Create an order or a line item
- Forecasting
- Quantity Max Available is only available in Forecasting for Sponsorship line item
- Forecast for multiple Priority:Sponsorship with total goal more than 100% is not accurate
- e.g. There're 150k single requests asking for 4 big boxes
- There're 10 and only 10 identical Priority:Sponsorship line items and each is eligible to show in any 4 big boxes with goal 100%
- Forecast shows each line item can get 150k impressions since each one is 100% goal. While each one should get 150k * 4 (big boxes) / 10 = 60k
- Forecasts shows each gets 150k*10% = 15k While each one should get the same 60k as calculated above
- In short, Priority:Sponsorship line item used in forecast wins every single request and the goal percentage is applied to become the total forecast. Which is wrong. Percentage for each one should be 4 * 10% = 40%
- e.g. There're 150k single requests asking for 4 big boxes
32.23. Benchmarks and Standards
- IAB
- https://www.bellmedia.ca/sales/digital/
- https://www.betterads.org/standards/
- Sizmek Benchmarks
- Based on 2016 H1 report, CTR (click-through rate = clicks / impressions) is around 0.14% in North America and South Asia is the highest 0.28% for standard banner
- Rich Media CTR is around from 6 to 8 times higher than standard banner
- Programmatic Rich Media that is not confined to the iFrame performs around 145% better than polite
- https://www.doubleclickbygoogle.com/insights/
- LiveIntent, AdButler, PowerInbox
32.24. User role
- Built-in
- administrator
- not Ad Exchange
- create/manage orders and run reports on orders they create
- create/manage/approve/cancel orders, edit targeting criteria and run reports on orders/sales/inventory
- create/edit orders, edit line items, upload creatives and run reports on orders and creatives
- run reports and evaluate the effectiveness of campaigns through read-only access to all functionality
- access to Ad Exchange funtionality, but not Ad Manager. Might not be available on your network
- view and edit users, roles, and teams, as well as access and accept the Ad Manager payment contract when completing the self-serve billing setup
33. XML
33.1. XML DOM
Everything is a node nodeType :: 1 - element, 2 - attribute, 3 - text, 4 - comment, 5 - document nodeName :: read-only, text node name is #text, document node name is #document nodeValue :: element node value is undefined
var elements = xmlDoc.getElementByTagName("title"); console.log(elements.length); var element = elements[0]; var elementToRemove = xmlDoc.getElementsByTagName('book')[0]; xmlDoc.documentElement.removeChild(y); element.parentNode.removeChild(element); // remove myself
xde = xmlDoc.documentElement; newNode = xmlDoc.createElement('book'); / create element node newTitle = xmlDoc.createElement('title'); newText = xmlDoc.createTextNode('A Notebook'); / create text node newTitle.appendChild(newText); / add text node to element node newNode.appendChild(newTitle); / add element node to element node
y = xmlDoc.getElementsByTagName('book')[0]; xde.replaceChild(newNode, y); / replace an element node aTextNode.replaceData(0,8,"Easy"); / replace data of text node (from start to 8 in length) // Use nodeValue, it's easier
var attributes = element.attributes; var attribute = element.getAttribute('lang'); var attributeNode = element.getAttributeNode('lang'); attributeNode.nodeValue; attributeNode.nodeValue = "new attribute value"; element.setAttribute('category', 'food');
element.removeAttributeNode(attributeNode); / remove an attribute node element.removeChild("category"); / remove an attribute node by name // remove multiple attribute nodes, you will have to loop all
var x = element.childNodes[0].nodeName;
33.1.1. Navigate
var fc = element.firstChild; var lc = element.lastChild; fc.parentNode; var nc = fc.nextSibling; nc.previousSibling; function get_nextSibling(n) { // Bypass empty text node while traversing var y = n.nextSibling; while (y.nodeType != 1) { y = y.nextSibling; } return y; }
33.2. Do not parse < and &
<![CDATA ]]>
33.3. Comment
<!-- Same as HTML but double -- is not allowed -->
33.4. XML newline is always LF
33.5. XML Element node name can contain -, _ and . but not space
33.6. Attribute value should be HTML entity encoded
Mainly enocde:
"=>">=>><=><
33.7. XML namespace
<root xmlns:h="http://www.w3.org/TR/html4/" xmlns:f="http://www.w3schools.com/furniture"> <h:table> <h:tr> <h:td>Apples</h:td> <h:td>Bananas</h:td> </h:tr> </h:table> <f:table> <f:name>African Coffee Table</f:name> <f:width>80</f:width> <f:length>120</f:length> </f:table> </root>
33.8. XSLT
XSLT stands for XSL (Extensible Stylesheet Language) Transformations XSLT uses XPath to navigate through elements.
XSLT code
<?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="/"> <!-- Associate the template with the root of the XML source document --> <!-- A template contains rules to apply when a specified node is match --> <h2>My CD Collection</h2> <table border="1"> <tr bgcolor="#9acd32"> <th style="text-align:left">Title</th> <th style="text-align:left">Artist</th> </tr> <xsl:for-each select="catalog/cd"> <!-- for-each loop, select contains XPath expression --> <!-- select="catelog/cd[artist='Bob Dylan']" --> <!-- =, !-, <, > --> <xsl:sort select="artist" /> <!-- sort --> <tr> <td><xsl:value-of select="title"/></td> <td><xsl:value-of select="artist"/></td> <xsl:if test="price > 10"> <td><xsl:value-of select="price"/></td> </xsl:if> <!-- choose, when, otherwise --> <xsl:choose> <xsl:when test="price < 10"> <td bgcolor=""><xsl:value-of select="price"/></td> </xsl:when> <!-- Multiple when's --> <xsl:otherwise> <td><xsl:value-of select="price"/></td> </xsl:otherwise> </xsl:choose> <!-- choose, when, otherwise. --> </tr> </xsl:for-each> <!-- Close a loop --> </table> </xsl:template> </xsl:stylesheet>
XML file
<?xml version="1.0" encoding="UTF-8"?> <catalog> <cd> <title>Empire Burlesque</title> <artist>Bob Dylan</artist> <country>USA</country> <company>Columbia</company> <price>10.90</price> <year>1985</year> </cd> <cd>...</cd> ... </catalog>
Template and subtemplates
<?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="/"> <html> <body> <xsl:apply-templates/> </body> </html> </xsl:template> <xsl:template match="cd"> <p> <xsl:apply-templates select="title"/> <xsl:apply-templates select="artist"/> </p> </xsl:template> <xsl:template match="title"> Title: <xsl:value-of select="."/> <br /> </xsl:template> <xsl:template match="title"> Artist: <xsl:value-of select="."/> <br /> </xsl:template> </xsl:stylesheet>
33.9. Invalid characters xml:invalid_characters
Characters ::  
change to version 1.1 or get rid of the characters
<?xml version="1.1" encoding="UTF-8" ?>
34. YAML
34.1. Basics
- Superset of JSON
- JSON
- Not supported
- comments
- (no term)
- Commas are used to separate fields
- curly brackets
- around objects
- double quotation marks
- around strings
- square brackets
- around arrays
- YAML
- Hyphens
- before arrays
- key
- need to double quoted the whole key if key has
:
- https://stackoverflow.com/a/21699210/2196360
- Multiline
- https://yaml-multiline.info/
34.2. Sample
--- # three dashes to separate documents # data types # - string :: single or double-quoted or no quote at all # - float # - integer # - boolean # - True, true, On, on, Yes, yes # - False, false, Off, off, No, no # - null :: `~` or `null` (unquoted) # Indentation # - Can be any number of spaces but tabs are not allowed doe: "a deer, a female deer" ray: a drop of golden sun stringWithLineFeed: "this is a line with linefeed \n" # Use >- or |- to remove linebreak at the end multipleLinedString: | paragraph 1 (keep newlines AKA literal, single newline at the end, AKA clip) paragraph 2 onelineStringButTypeInMultipleLines: > paragraph 1 (replace newlines with spaces AKA folded) paragraph 2 multipleLinedStringWithStrip: |- paragraph 1 (strip the trailing newlines) paragraph 2 multipleLinedStringWithoutStrip: |+ paragraph 1 (keep trailing newlines AKS keep) paragraph 2 pi: 3.14159 xmas: true french-hens: 3 calling-birds: - huey - dewey - louie - fred anotherArray: [ 1, 2, 3, 4, 5 ] anotherArray2: [ "one", "two", "three", "four" ] anotherArray3: # array of objects - type: file path: /path/to/file1.log - type: file path: /path/to/file2.log xmas-fifth-day: # object or YAML calls it dictionary calling-birds: four french-hens: 3 golden-rings: 5 partridges: count: 1 location: "a pear tree" turtle-doves: two inlineDictionary: { thing1: huey, thing2: louie, thing3: dewey } # in abbreviated form
35. OpenAPI
- Schema validator (validate schema)
- Editor
- For OpenAPI 3.0 https://editor.swagger.io/
- Editor Next
- For partial OpenAPI 3.1 https://editor-next.swagger.io/
- (no term)
- Postman > APIs > API definition
- Specifications
- https://oai.github.io/Documentation/specification.html
- Implementations e.g. Validators, Documentation (HTML)
- https://json-schema.org/implementations.html
- Redoc (React JSON Schema to HTML)
35.1. Data Types
typeandformat- https://spec.openapis.org/oas/v3.0.3#data-types
- Extension of https://datatracker.ietf.org/doc/html/draft-wright-json-schema-00#section-4.2
- null
- not supported in OpenAPI. Use
nullable: true - (no term)
- boolean
- (no term)
- object
- (no term)
- array
- (no term)
- string
- (no term)
- number
- More
- integer
- Extension of https://datatracker.ietf.org/doc/html/draft-wright-json-schema-00#section-4.2
35.1.1. array
- Unlike general JSON Schema,
itemskeyword is required An array of any type
# [ "hello", -2, true, [5.7], {"id": 5} ] type: array items: {}
An array of 2 types
# ["foo", 5, -2, "bar"] type: array items: oneOf: - type: string - type: integer
An array of an array
# [ [1, 2], [3, 4] ] type: array items: type: array items: type: integer
An array of an object
# [ {"id": 5}, {"id": 8} ] type: array items: type: object properties: id: type: integer
# Array of Pets type: array items: $ref: '#/components/schemas/Pet'
An array's length
# Without minItems, an empty array is valid type: array items: type: integer minItems: 1 maxItems: 10
Unique items
type: array items: type: integer uniqueItems: true # [1, 2, 3] – valid # [1, 1, 3] – not valid # [ ] – valid
35.1.2. Markdown
35.2. schema - reuse properties
$refin- requestBody
- schema
- parameters
- allOf
allOfin- schema
paths: '/customers': post: tags: - Customer requestBody: $ref: '#/components/requestBodies/CreateCustomer' responses: 201: description: Created content: application/json: schema: $ref: '#/components/schemas/CustomerId' get: tags: - Customer responses: 200: description: OK content: application/json: schema: type: array items: $ref: '#/components/schemas/Customer' /customers/{CustomerId}: get: tags: - Customer parameters: - $ref: '#/components/parameters/CustomerId' responses: 200: description: OK content: application/json: schema: allOf: - $ref: '#/components/schemas/CustomerProperties' - $ref: '#/components/schemas/CustomerRequiredProperties' put: tags: - Customer requestBody: $ref: '#/components/requestBodies/CreateCustomer' parameters: - $ref: '#/components/parameters/CustomerId' responses: 204: description: Updated patch: tags: - Customer requestBody: description: Update customer with properties to be changed content: application/json: schema: allOf: - $ref: '#/components/schemas/CustomerProperties' - type: object properties: Segment: nullable: true parameters: - $ref: '#/components/parameters/CustomerId' responses: 204: description: Updated components: schemas: CustomerProperties: type: object properties: FirstName: type: string LastName: type: string DOB: type: string format: date-time Segment: type: string enum: - Young - MiddleAged - Old - Moribund CustomerRequiredProperties: type: object required: - FirstName - LastName - DOB Customer: allOf: - $ref: '#/components/schemas/CustomerId' - $ref: '#/components/schemas/CustomerProperties' - $ref: '#/components/schemas/CustomerRequiredProperties' requestBodies: CreateCustomer: description: Create a new customer content: application/json: schema: allOf: - $ref: '#/components/schemas/CustomerProperties' - $ref: '#/components/schemas/CustomerRequiredProperties'
37. Software Programming
37.1. Programming Paradigm
- A way to classify programming languages based on their features. A language can have multiple paradigms
- Which execution model? Behavior of elements of the language
- e.g. grouping a code into units along with the state that is modified by the code
- Style of syntax and grammar
37.1.1. Imperative programming
- Imperative programs spend lines of code describing the specific steps (flow control) used to achieve the desired results: How to do things
- The steps mutate the program's state
37.1.2. Declarative programming
- Declarative programs abstract the flow control process, and instead spend lines of code describing the data flow: What to do. The how gets abstracted away
- To minimize or eliminate side effects
- Examples
- Functional Programming
- SQL
- In a way, declarative programming is a layer of abstraction on top of imperative programming
- Core concepts
- Immutability
- Separating functions and data
- functions are treated like any other variable e.g. can be passed as an argument to other functions, returned by another function, be assigned as a value to a variable
- function that does at least one of the following
- take one or more functions as arguments
- return a function as its result
- To achieve e.g. strategy pattern
Partial Application (Currying)
$add = fn($x, $y, $z) => $x + $y + $z; $add_partial = fn($x) => fn($y) => fn($z) => $add($x, $y, $z); // some code later on $add_5 = $add_partial(5); // the value for the first parameter is confirmed // some code later on $add_5_and_6 = $add_5(6); // the value for the second parameter is confirmed $total = $add_5_and_6(7); // eq. to $add_partial(5)(6)(7)
- To achieve procedural operations, or constructor or method injection
- Recursion
- As the primary and only means of looping
- Function Composition
- Provide other functions to a composed function, running the composed function will run these functions in series e.g. separate first/last names from full name, add initial based on first/last names. Chaining of these functions
- Functional Programming generally does not have if statements, loop constructs etc.
- lets us decompose a data structure by its shape instead of its content
- To achieve mediator pattern
37.1.3. Functional Programming (FP)
- See Declarative Programming
- FP is the process of building software by composing pure functions, avoiding shared state, mutable data, and side-effects
- Functional code tends to be more concise, more predictable, and easier to test than imperative or object oriented code
- An immutable object is an object that can’t be modified after it’s created. Conversely, a mutable object is any object which can be modified after it’s created.
- Pure vs impure functions. A pure function must satisfy both of:
- Referential transparency
- the function always gives the same return value for the same arguments. This means that the function cannot depend on any mutable state
a(b(c(x)))- the function cannnot cause any side effects which may include I/O (console.log), modifyng a mutable object, reassigning a variable, etc.
- Don't trigger any external process
- Monad
- VS OOP
- FP is better at
- ensuring stateless code
- heavy duty data manipulation
- more terse
- OOP is better at
- managing state in general
- handling interface interactions
- more verbose
- FP is better at
37.1.3.1. Avoid conditions
Guard clauses or early returns
function abc() { if (...) { return 'special case #1'; } if (...) { return 'special case #2'; } return 'default'; }
37.1.3.2. Avoid reassigning variables
- Incremental computations
const VIDEO_VALIDATIONS = [ { // Must provide either both a height + width, or neither isValid: video => validateHeightWidthConsistency(video.videoFiles), message: ERROR_MESSAGES.InconsistentWidthHeight }, { // Must have ONE OF either a file or a URL isValid: video => validateVideoFileAndUrl(video.videoFiles), message: ERROR_MESSAGES.InvalidVideoFiles }, { // Video URL must be a valid link isValid: video => validateVideoURL(video.videoFiles), message: ERROR_MESSAGES.InvalidVideoURL }, { // Title cannot be blank isValid: video => !!video[INPUT_TYPES.Title], message: ERROR_MESSAGES.BlankTitle }, { // ID must be alphanumeric isValid: video => video[INPUT_TYPES.Id].match(ID_PATTERN) !== null, message: ERROR_MESSAGES.InvalidId } ]; const validateVideo = video => { return VIDEO_VALIDATIONS.map(({isValid, message}) => isValid(video) ? undefined : message ).filter(Boolean); }; const printVideoErrors = video => { console.log(validateVideo(video).join('\n')); };
- Building complex objects
const hasDateRange = dateRangeFrom && dateRangeTo; const queryValues = { sortBy: sortField, orderDesc: sortDirection === SORT_DESCENDING, words: query, from: hasDateRange && format(dateRangeFrom.setHours(0, 0, 0, 0), DATE_FORMAT), to: hasDateRange && format(dateRangeTo.setHours(23, 59, 59), DATE_FORMAT) };
- Indeterminate loops
Reassignments aren’t pure evil and exterminating all of them won’t make your code better. They are more like signs: if you see a reassignment, ask yourself if rewriting the code without it would make it more readable. There’s no right or wrong answer, but if you do use a reassignment, isolate it in a small function, where it’s clear what the current value of a variable is.
37.2. Side Effect
An operation, function or expression is said to have a side effect if it modifies some state variable values outside its local environment, that is to say has an observable effect besides returning a value (the main effect) to the invoker of the operation
37.3. DevOps
- Methodologies
- People over process over tools
- Continuous delivery
- Lean Management
- Work in small batches
- Work in progress limits
- Feedback loops
- Visualization
- Visible ops change control
- Eliminate fragile artifacts
- Crate a repeatable build process
- Manage dependencies
- Create an environment of continous improvement
- Infrastructure as code
- Systems treated like code
- Checked into source control
- Reviewed, built and tested
- Managed programmatically
- 10 Best practices
- Chaos Monkey
- Blue/Green Deployment
- Dependency Injection
- Andon Cords (anyone can pull to stop the production line)
- The Cloud (fast API services)
- Embedded Teams
- Blameless Postmortems
- The people whose actions have contributed to an accident can give a detailed account of 5 W's
- And they can give this detailed account without fear of punishment
- The detailed account gives an understanding of the mechanism, pathology, and operation of the failure
- The account also guarantees that it will repeat
- It's a meeting within 48 hours of the incident
- Have a third party run it
- Public Status Page
- Developers on Call
- Incident Command System
- Glossary
- Provision
- Making a server ready for operation, including hardware, OS, system services, network connectivity
- Deployment
- automatically deploying and upgrading applications on a server
- Orchestration
- performing coordinated operations across multiple systems
- Configuration management
- management of change control for system configuration after initial provision; maintaining and upgrading the application and application dependencies
- Imperative/procedural
- commands necessary to produce a desired state are defined and executed
- Declarative/functional
- a desired state is defined, relying on the tool to configure a system to match that state
- Idempotent
- the ability to execute repeatedly, resulting in the same outcome
- Self service
- the ability for an end user to initiate a process without having to go through other people
- Continuous Integration
is the practice of automatically building and unit testing the entire application frequently. Ideally, every source code check in
- CI Phases and corresponding tools
- Version control
- CI systems (Jenkin, CloudBees, Bamboo, CircleCI, TravisCI)
- Build (Make/Rake, Maven, Gulp)
- Test (JUnit, golint/gofmt, RuboCop, Robot, Protractor, Cucumber, Selenium, Sauce Labs, ApacheBench, JMeter, Brakeman, Veracode)
- Artifact repository (Artifactory, Nexus, Docker Hub, AWS S3)
- Deployment (Rundeck, UrbanCode, ThoughtWorks, Deployinator)
- Docker Swarm, Google Kubernetes, Apache Mesos
- Continuous Delivery
- is the addtional practice of deploying every change to to a production like environment and performing automated integration and acceptance testing
- Continuous Deployment
- extends this to where every change goes through full enough automated testing
37.4. Visual Regression Testing
37.4.1. Tools
https://github.com/mojoaxel/awesome-regression-testing https://www.creativebloq.com/features/the-5-best-visual-regression-testing-tools
Headless browser
- PhantomJS
- WebKit
- SlimerJS
- Firefox
CasperJS :: JavaScript navigation scripting and testing utility for PhantomJS and SlimerJS ResemberJS :: JavaScript / HTML5 library for making image comparisons.
BackstopJS 3
37.5. Security Scans
- Tinfoil Security
- SiteImprove
37.6. Performance Testing
37.7. Code Quality
37.7.1. Cyclomatic Complexity
- Number of decision points in a method plus one for the method entry
- Decision points
- if, while, for and switch's
caseanddefault - Not including
else, class defintion
- if, while, for and switch's
- 1 to 4
- 5 to 7
- 8 to 10
- 11+
37.7.2. TTD: refactor legacy code
- Add pin-down tests to cover 100% legacy code
- Refactor
- Start from small refactoring e.g. convert string, numbers and other hard coded variables to constants
- De Morgan's Laws
- Catalog of refactorings
- usually after refactoring, a new feature needs to be implemented
- Add small test and implement smallest incremental amount of code to pass the test
37.7.3. Code Climate
- Free for open source projects on GitHub
- https://docs.codeclimate.com/docs/advanced-configuration
37.7.4. Code Scene
- CodeScene.io identifies hotspot files that generate churns
37.7.5. SonarCloud.io
- Cloud version of the on-premise version SonarQube
- Visualize duplication
37.7.6. Snyk
- Check security for dependencies used in code
37.7.7. De Morgan's Laws
!a && !beq. to!( a || b )!a || !beq. to!( a && b )
37.7.8. S.O.L.I.D. - Solid Principles
37.7.8.1. Single responsibility :: A class should have one, and only one reason to change
37.7.8.2. Open/closed principle
- Entities should be open for extension, but closed for modification
- Open to extension means adding subclasses as needed
- Closed to modification avoids "tweaking" the code to handle new situations
- Example
- Before
- rectangle, circle, areaCalculate
- Should be
- interface:areaCalculate, then rectangle and circle implement this interface
- In short, we should create abstract or parent classes or implement dependency injection or Composition
37.7.8.3. Liskov substitution principle
- Subclass/derived class should be substitutable for their base/parent class
- Constrains subclass design
- Helps programmers design good polymorphism
- Subclasses shared methods in base/parent class should have the same type of input and output
- In short, the method signature has to be the same
- The constructor could be different
37.7.8.4. Interface segregation principle
- A Client should not be forced to implement an interface that it doesn’t use
- A client should depend on the smallest set of interface features: the fewest methods and attributes
- Example
- Before
- WorkerInterface{ work, sleep }, HumanWorker, RobotWorker (Robot implements an empty sleep method)
- Should be
- WorkAbleInterface{ work } , SleepAbleInterface{ sleep }, HumanWorker, RobotWorker
37.7.8.5. Dependency inversion principle (DIP)
- High-level modules should not depend on low-level modules. Both should depend on abstractions
- Abstractions should not depend on details. Details should depend on abstractions
- Just changing the dependency module and High-level module will not be affected by any changes to the Low-level module
- A direct dependency, on a concrete class, needs to be "inverted", which means to refer to an abstraction
- Depend on abstraction classes or interfaces
- Avoid concrete class name dependencies
- Example
- Before
- MySQLConnection, PasswordReminder{ __construct(MySQLConnection $dbConnection) }
- Should be
- ConnectionInterface, MySQLConnection imp. ConnectionInterface, PasswordReminder{ __construct(ConnectionInterface $dbConnection) }
- In short, it means when we do dependency injection, use the parent Interface and not a specific type
- Inversion of Control (IoC)
- Instead of the main function (the app) is fully responsible to sequentially instantiate and call methods, the app is now only responsible to do the binding and instantiation of dependencies, the specific methods are called.
- Dependence Injection (DI)
- A software design technique in which the creation and binding of dependencies are done outside of the dependent class
- Separates the construction and configuration of services from its usage, relieving the dependent class from the former responsibility, decoupling it from its dependencies, and improving reusability and ease to test
- 3 ways
- Constructor injection
- when dependencies are provided through the dependent class' constructor
- Interface injection
- when dependencies are provided directly into a dependent class method as an argument
- Setter injection
- when dependencies are provided via a public property of the dependent class
- A software design technique in which the creation and binding of dependencies are done outside of the dependent class
37.7.9. General responsibility assignment software principles (GRASP)
37.8. Drupal
DRUPALVM.com devwithlando.io
Send metrics and logs (e.g. Watchdog) to Elasticsearch
Kibana PHP APM (Application Performance Metrics)
37.9. Machine Learning
37.9.1. Supervised vs Unsupervised Learning
- Supervised is to label data at first
- given sample data (training set) of X/Y pairs (X is input or feature and Y is output or target or true value), now given X', to predict Y \(\hat{y}\)
- Examples
- Regression
- Both X and Y are a lot or continuous
- First training example
- (x1, y1) = (123, 321)
- ith training example
- (xi, yi) = (321, 321)
- Classification
- Y is finite, X could be a lot e.g. classify known dogs into 5 categories
- Put new observations into existing criterion categories based on matches with existing data
- Observations are labeled with (intended) outcome
- Label may be subjective e.g. good book vs bad book categories
- then predict outcome category with other variables
- then apply model to new data
- Accuracy is important in supervised learning
- To avoid biase, sampling data should cover all possible data in all categories (diversities)
- Focus on small categories and edge cases
- Label could be biased e.g. good book vs bad book. To limit bias, include labels from a wider range of judges
- multiple people are making a judgment, do they reach the same conclusion?
- Anchoring bias
- first impressions or judgments are strong and resistant to new info
- Reflected on AI, once a model is developed, new variables for each case may not have as much influence as they should
- Stability bias
- the model makes consistent categorizations even when the outcome has fundamentally shifted
- e.g. impressions of a celebrity may drastically shift in public perception
- Occam's Razaor
- One should not multiply entities beyond necessity
- (no term)
- The simplest explanation is usually the correct one
- It's limited to existing variables and categories in original data. Cannot extrapolate
- K-NN
- decision tree
- regression
- Examples
- (no term)
- Unsupervised is let the machine to cluster the data
- Group cases based on similarity of measured attributes without specific criterion
- Data only comes with inputs x, but not output labels y. Algorithm has to find structure in the data
- Examples
- Clustering algorithm
- group similar data points together
- Anomaly detection
- find unusual data points
- Dimensionality reduction
- compress data using fewer numbers
- K-mean clustering e.g. when there are lots of unlabeled data
- Group cases based on similarity of measured attributes without specific criterion
37.9.2. Causes of classification errors
- Bayes' Theorem
- True positive
- people should be put into a particular category and what % of those people out of total
- False positive
- people should not be put into a particular category and what % of those people
- (no term)
- The base rate of the outcome
37.9.3. Decision Tree
- Predictors
- Sky
- sunny, overcast, rain
- Weekend
- weekday, weekend
- Wind
- low, high
- Outcome
- Li goes to the beach or not
- Decision Tree
Sky
- Sunny
- 2 yes, 2 no, high/low wind, does not matter
- weekday
- 0 yes, 2 no
- weekend
- 2 yes, 0 no
- Overcast
- 1 yes 3 no, weekday or weekend, does not matter
- high wind
- 0 yes 2 no
- low wind
- 1 yes 1 no
- Rain
- 0 yes, 4 no
- (no term)
- no other leaf nodes, high/low wind and weekday/weekend, do not matter
37.9.4. K-nearest neighbor - K-NN
- Preparation
- classify 1000 known dogs into 5 categories, and creat 2 predictors: hair length and weight
- Given a unknown dog with a hair and weight, try to classify it
- Put the unknown dog into a chart with hair and weight, choose a K value of 5, which means find the 5 closest known dogs
- Let's say 2 of the 5 closest neighbors are Husky, 3 are Shepherd. The prediction for the dog might be Shepherd
37.9.5. K-mean clustering
37.9.6. Naive Bayes
- Predictors
- Hair, height, weight
- Given a dog with those 3 metrics, try to classify the dog into one of the 3 categories Terrier, Hound or Sport
Treat each predictor independently e.g. given only the hair color, try to categorize the dog
- Hair of brown color is 40% Terrier, 10% Hound and 50% Sport. When doing this, ignore other predictors' values
- Height of 50cm is 20% Terrier, 10% Hound, 70% Sport
- Weight of 10lb is 10% Terrier, 5% Hound and 85% Sport
- Add a weighted multiplier for each predictor e.g. 3 for Hair, 2 for Height and 1 for Weight
- Hence, a dog with those 3 metrics, being a
- Terrier is 40%*3 + 20%*2 + 10%*1 = 170%
- Hound is 10%*3 + 10%*2 + 5%*1 = 55%
- Sport is 50%*3 + 70%*2 + 85%*1 = 375%
- Hence, the prediction is the dog is a Sport dog
- Good for binary or multiclass classification
37.9.7. Bias vs Variance
- Bias is the gap between the predicted value and the actual outcome
- Variance is when the predicted values are scattered all over the place
- High bias and low variance
- consistently wrong
- High bias and high variance
- consistenly wrong in an inconsistent way
37.9.8. Signal vs Noise, underfitting and overfitting
- Signal is something that can be used to make accurate predictions
- Noise is natural variances in the data that might not offer any insights
- When the model has noise predictors, it includes other unnecessary factors, and it is called overfitting
- Underfitting is some signal predictors are not and should be included in the model
37.10. Ansible
37.11. Licensing software:license
- The software code that can be viewed and editted is subject of copyright
- What the software does when it executes is the subject of patent
- License copyrights, trademarks and patents
37.11.1. Open Source
- https://opensource.org/ Open-Source Initiative Criteria
- Free redistribution
- Must provide source code or make it available
- Must allow derived works
- The itegrity of author's code must be maintained
- No discrimination against persons or groups
- No discrimination against fields of endeavor
- Distribution of only one license required to secure rights
- License must not be specific to product
- License must not restrict other software
- License must be technically neutral
- try to affiliate with an open-source foundation like Apache, Mozilla or Outercurve
- Create a business entity for succession plan
- BSD (Berkeley Software Distribution)
- GPL and other GNU licenses (General Public License)
- All GNU licenses
- http://www.gnu.org/licenses/
- GPL
- http://www.gnu.org/licenses/#GPL
- MIT license
- Permissive
- If you don't care about restrictions or what happens downstream
- Apache
- If you want to encourage commercial adoption of your code
- Only it has an implicit software:license:cla
- Copyright vs Copyleft
- Copyright
- I give right to someone
- Copyleft
I give right to someone so that he/she has the same right
- The same license as the original code must be maintained
- GPL v3
- Combining GPL covered source code with non-GPL code does make target code subject to GPL
- Weak Copyleft
- LGPL v3
- Distribution of binary file libraries (DLLs) does not affect the license of target code that references DLLs
- Permissive
- Few restrictions and obligations. Non-Copyleft. Software covered by permissive licenses can be included in copyleft covered software but not the other way around
- e.g. BSD, MIT, Apache
37.11.2. Creative Commons
- https://creativecommons.org/
- for content (non-software), embrace copyleft principles
- (no term)
- Concepts
- Attribution
- author, creator
- (no term)
- Restrictions
- Share Alike
- Noncommercial
- No Derivatives
37.11.3. Contributor License Agreements (CLA) software:license:cla
38. Tools
38.1. MacOS
38.1.1. Versions
- Catalina
- 10.15
38.1.2. Escape key not working
- Activity Monitor > Find Siri > Force Quit > Start again
38.1.3. Screenshot and Screen Recording ⇧ ⌘ 5
- Recording
- Stop
⌃ ⌘ Esc- Options
- turn on microphone
Cmd-S-4- Copy portion of screen
Cmd-C-S-4
Cmd-S-3- Copy entire screen
Cmd-C-S-3
38.1.4. Directories
- ~/Library
- LaunchAgents or LaunchDaemons
- for scheduled or background tasks
- (no term)
- Preferences or
/Library/Preferences
38.1.5. Shortcuts
- https://support.apple.com/en-ca/HT201236
- System Preferences > Keyboard > Shortcuts
- To disable
- Services
⌘ ⇧ a- Mission Control
F6- Quick Note
Fn-q
- To add
- App Shortcuts
- Google Chrome.app
- Move Tab to New Window (added)
⇧ ⌘ w
- To disable
⌥ Brightness Up or Down⌥ Volume up or down- Menu bar
- Ventura
⌃ F1(enable keyboard access) then⌃ F2- Monterey
Fn-m
Fn-aFn-hFn-nFn-cFn-dFn-eFn-fFn-q⌥ ⌘ Esc
38.1.5.1. Finder
- Jump to Home folder of the current macOS user account
⇧ ⌘ h- Create a new folder
⇧ ⌘ n- View the Finder items as icons
⌘ 1- View the Finder items as list
⌘ 2- Use Quick Look to preview the selected items
⌘ y- Open the parent folder
⌘ Up- Show hidden items
⇧ ⌘ .- Go to
⇧ ⌘ g- Go to next tab
⌃ Tab- Permanent delete
⌘ ⌥ Deletefollowed by⌘ dto confirm- Search files
⌘ f- Saved Search
.savedSearchis XML...icon on the top toShow Search Criteria, change view mode and grouping, thenSave
- (no term)
- Tags
- Caveat
- For mounted image file
- Both system and custom tags cannot be searched but can be displayed using Group by tags
- Custom tags cannot be searched and displayed using Group by tags for Saved Searches
- Custom tags with a color can be displayed using Group by tags, but only the color names are displayed
- For mounted image file
- Caveat
- Kind
- It's not the file extension.
Get infoand look forKind - Name
ends withfor file extension. Case-insensitive- Raw Query
- regex
- Syntax
- File Metadata Attributes Reference
- Sportlight Metadata and iCloud Metadata attributes
- ">file name
"yourRegexPattern"c- case-insensitive
- Syntax
38.1.5.2. Window
- Command + `
- switch in-app windows e.g. chrome normal and chrome incognito
- Ctrl + Tab
- switch in-app tab e.g. chrome tabs
- Close current window
Cmd-w- Close all windows of the app
Cmd-M-w- (no term)
- No shortcut for maximize current window
- Switch to next window of the app
Cmd-`- Hide current app
Cmd-hand bring it backCmd-Tab- Hide all windows of other apps
Cmd-M-h- Bring back minimized app
Cmd-Tabto the app, before releasingCmd, pressAlt
38.1.5.3. Input
- Emoji
- Command + Control + Space
- fn
- switch between Input Source
- Go to file top
Command + Fn + LeftorCommand + Up- Go to file bottom
Command + Fn + RightorCommand + Down- Move to previous / next word
- Option + Left / Right
- Move to beginning / end of a line
- Command + Left / Right
- Page Down / Up
Fn + Down / Up
38.1.5.4. Notes
38.1.6. Homebrew
38.1.6.1. Install :: https://brew.sh/
38.1.6.2. Install brew version of zsh
# Current Login Shell # System Pref > Users & Groups > Unlock to make changes > right click on your name > Advanced Options > Login shell # default is /bin/zsh that is also what comes with MacOS dscl . -read /Users/$USER UserShell # should be # UserShell: /bin/zsh brew install zsh which zsh # should be # /opt/homebrew/bin/zsh -> ../Cellar/zsh/5.8_1/bin/zsh # Change the current user to use this as the Login Shell sudo dscl . -create /Users/$USER UserShell /opt/homebrew/bin/zsh # restart Terminal, verify the new shell is /opt/homebrew/bin/zsh dscl . -read /Users/$USER UserShell # should be /opt/homebrew/bin/zsh echo $SHELL # should be -zsh echo $0 # done!
38.1.6.3. Basics
# like sudo apt-get update, it also updates brew and cask brew update # list installed formulas/formulae. A formula is a package definition or a package. brew list # show an installed package's installation path brew list git # list installed applications brew list --cask # see what versions you have, and later switch to a different version brew list --versions git # Change versions brew switch git 2.5.0 # install an application using dmg brew install --cask firefox # upgrade a package/formula brew upgrade git # upgrade all packages/formula brew upgrade # List all added/tapped repositories. Third-Party Repositories = Tap brew tap # brew folder/directory which is /opt/homebrew brew --repository # Clones `https://github.com/<user>/homebrew-<repo>` to /opt/homebrew/Library/Taps, later formulae will be updated when `brew update` is run brew tap <user/repo> # Remove the tap # Check a package versions, caveats, etc. May do it before installation brew info git
38.1.6.4. KeyCastr
https://github.com/keycastr/keycastr#readme
brew install --cask keycastr
38.1.6.5. telnet
brew install telnet telnet towel.blinkenlights.nl
38.1.6.6. mutagen
brew tap mutagen-io/mutagen brew install mutagen-io/mutagen/mutagen brew install mutagen-io/mutagen/mutagen-compose # /opt/homebrew/Cellar/mutagen/0.13.1
38.1.6.7. svn
brew install svn
svntools have been installed to: /opt/homebrew/opt/subversion/libexec The perl bindings are located in various subdirectories of: /opt/homebrew/opt/subversion/lib/perl5 You may need to link the Java bindings into the Java Extensions folder: sudo mkdir -p /Library/Java/Extensions sudo ln -s /opt/homebrew/lib/libsvnjavahl-1.dylib /Library/Java/Extensions/libsvnjavahl-1.dylib ==> Summary 🍺 /opt/homebrew/Cellar/subversion/1.14.1_4: 234 files, 33.1MB
38.1.7. Display
- 3024 × 1964
- MacBook Pro (14-inch, 2021, M1 Pro)
- window.screen.width
- 1512
- window.screen.height
- 982
- window.screen.devicePixelRatio
- 2
38.1.8. Terminal
open afile,open adir,open .,open ..- open in Finder
open -a TextEdit afile.txt- open a file using an app
ls -al | open -f- redirect stdout to TextEdit
cat a.tsv | open -fa Numbers- redirect stdout to an app
- (no term)
- Clipboard
ls -la | pbcopy,pbcopy < afile.txtpbpaste > afile.txt4 different clipboards!
echo "first" | pbcopy -pboard general echo "second" | pbcopy -pboard find pbpast -pboard find
- (no term)
Screen capture
screencapture /path/to/screencapture.png # interactive screencapture -i /path/to/screencapture.png # -m main monitor only. Default all monitors # -t, format: png, pdf, jpg, tiff # -C, show cursor # -T, delay in seconds # -P, open file with Preview
Cmd-S-3- interactive
- (no term)
sudo shutdown- Options
h- halt
r- reboot
s- sleep
- Time
- now
sudo shutdown -h now- (no term)
- +minutes
- (no term)
- yymmddhhmm
- Options
38.1.10. NFS
- Give Full Disk Access to nfsd
- System Preferences > Security & Privacy > Privacy > on the left
Full Disk Access, add another Cmd-S-gthen type/sbin/nfsd
- System Preferences > Security & Privacy > Privacy > on the left
sudo nano /etc/exports # where $U is `id -u`, $G is `id -g` # Since Catalina, /Users are in another volume # instead of doing this: # /Users -alldirs -mapall=$U:$G localhost # now insert the following # /System/Volumes/Data -alldirs -mapall=$U:$G localhost # more about /etc/exports file # http://scottlab.ucsc.edu/xtal/wiki/index.php//etc/exports sudo nano /etc/nfs.conf # insert # nfs.server.mount.require_resv_port = 0 # more about /etc/nfs.conf file # http://scottlab.ucsc.edu/xtal/wiki/index.php//etc/nfs.conf sudo nfsd restart
38.1.12. Photos app
- Repair a photo library
- In
/Applicationsfolder,⌥ ⌘double clickPhotos.app - Shared albums will be re-downloaded
- In
- Create and open a photo library and set it to the System Photo Library
- In
/Applicationsfolder,⌥double clickPhotos.app
- In
38.1.13. Time Machine
- Disk Utility > Select all Devices and select the external hard drive > Erase
- Eject external drive fail
- Go to Finder then force quit
- ⌘ ⌥ Escape
- (no term)
- Or logout Mac and login back
38.1.14. Disk Utility
38.1.14.1. Create a blank image
- Format
- APFS or APFS (Case-sensitive)
- For the latest Mac that uses SSD
- Mac OS Extended (Journaled)
- For macOS 10.12 or earlier
- FAT or ExFAT
- For Windows. For less than 32 GB, use FAT
- assign a passphrase to access the created image file
- Partition (aka Scheme / Partition Map)
- GUID Partition Map (GPT)
- All Intel-based and Apple silicon Mac computers. New Microsoft Windows-based computers can use this scheme
- Master Boot Record
- For all Microsoft Windows-based computers
- Apple Partition Map
- For old PowerPC-based Mac computers
- Image Format
- Sparse Bundle Disk Image
- .sparsebundle file extension
- Creates / has a lot of files
- Use Sparse Bundle for macOS over 10.5
- Sparse Image
- .sparsebundle file extension
- For earlier version of macOS
- Read/write disk image
- .dmg file extension
- Change the .dmg capacity size later
- Inject the disk then Disk Utility app > Images > Resize
- The app can't resize .dmg in APFS format, use command line. Inject disk before running
hdiutil resize -size 25G /PATH/TO/DISK/IMAGE.dmg- Mount it and you will see the Capacity size is set
- Expand the Container and the volume to verify the size matches
- The app can't resize .dmg in APFS format, use command line. Inject disk before running
- Inject the disk then Disk Utility app > Images > Resize
- Change the .dmg capacity size later
38.1.15. Other Apps
38.1.15.1. XnViewMP
~/.config/xnviewmpdirectory is created to store settings (xnview.ini), catalog (XnView.db) and thumbnails (Thumb.db)
38.1.16. Logitech MX Master 3S
# reboot MacBook after running sudo defaults write /Library/Preferences/com.apple.airport.bt.plist bluetoothCoexMgmt Hybrid # to revert sudo defaults delete /Library/Preferences/com.apple.airport.bt.plist bluetoothCoexMgmt
38.2. PhpStorm
38.2.1. Installation and 64 bit
- Just install the new version and it will guide you to uninstall and keep the settings.
- For activation, choose License server, address is http://idea.lanyus.com/
- Used before
- https://www.cnblogs.com/oucbl/p/11664610.html, http://idea.qinxi1992.cn/
- From 2019.1.2
- Just
0.0.0.0 https://account.jetbrains.com:443in/etc/hostsand linux:dns:flush - Before
0.0.0.0 account.jetbrains.com0.0.0.0 www.jetbrains.comin/etc/hosts
- Help > Check for Update > Change setting to don't auto check update
- Still can update plugins manually in this way
- Install Java SE Development Kit (JDK) 64 bit
Open
C:\Program Files (x86)\JetBrains\PhpStorm 10.0.3\bin\PhpStorm64.exe, Help > About, you should seeJRE: amd64 JVM: 64-bit
If not, set environment variable
JAVA_HOMEtoC:\Program Files\Java\jdk1.8.0_101and restartPhpStorm64.exe- Restart
PhpStorm64.exeand open multiple projects to test RAM - Finally, edit the shortcut to use
PhpStorm64.exe
- Open a file from command line
MacOS
- Create
~/phpstorm
#!/bin/sh open -na "PhpStorm.app" --args "$@"
chmod ~/phpstorm sudo ln -s ~/phpstorm /usr/local/bin/phpstorm
- Create
- Installation directories
- ~/Library/Application Support/JetBrains/PhpStorm2022.3
- Cache and history files
- ~/Library/Caches/JetBrains/PhpStorm2022.3
- Plugins
- ~/Library/Application Support/JetBrains/PhpStorm2022.3/plugins
- Logs
- ~/Library/Logs/JetBrains/PhpStorm2022.3
38.2.2. Material Theme UI
- https://github.com/ChrisRM/material-theme-jetbrains
- Settings > Appearance & Behavior > Use custom font
- Settings > Appearance & Behavior > Material Theme > Project View
38.2.3. Change config directories
- https://intellij-support.jetbrains.com/hc/en-us/articles/207240985-Changing-IDE-default-directories-used-for-config-plugins-and-caches-storage
- Before doing that, export settings so that you can import them later
Help | Edit Custom Properties
idea.config.path=c:/work/idea/caches/trunk-config idea.system.path=c:/work/idea/caches/trunk-system idea.plugins.path=c:/work/idea/caches/trunk-plugins
- Windows config path
- C:\Users\Li\AppData\Roaming\JetBrains\PHPStorm2020.1
- codestyles
- C:\Users\Li\AppData\Roaming\JetBrains\PHPStorm2020.1
- Windows installation path
- C:\Program Files\JetBrains\PhpStorm 2020.1\bin
38.2.4. Open big file
Ensure phpStorm is running in 64bit.
https://intellij-support.jetbrains.com/hc/en-us/articles/206544519 This is called idea.config.path Create a custom idea.properties file in one these locations. This is called
- For Windows
- in %USERPROFILE%\.IntelliJIdeaXX or %USERPROFILE%\.IdeaICXX,
- For Windows
- actually %USERPROFILE%\.PhpStorm2017.1\config
- For *NIX
- in ~/.IntelliJIdeaXX or ~/.IdeaICXX
- For macOS
- in ~/Library/Preferences/IntelliJIdeaXX or ~/Library/Preferences/IdeaICXX
Don't change the system idea.properties at install-path/bin
Help > Edit Custom Properties to create it and edit
# Changing this will affect global search performance! # in kb, it's 500mb # Provide code assistance idea.max.intellisense.filesize=500000 # Open file size idea.max.content.load.filesize=500000
Help > Edit Custom VM Options
-Xmx8192m # old :: change ~-Xms~ and ~-Xmx~ to ~-Xms2g~ and ~-Xmx1g~
More settings :: http://www.jetbrains.com/help/idea/tuning-the-ide.html
38.2.4.1. Plugin: JB SDK Bintray Downloader
Install JetBrains Plugin Find Action > Get JB SDK Select the most recent version to download and then install: e.g. jbsdk8u152b941_windows_x64.tar.gz %USERPROFILE%\.PhpStorm2017.1\config\jdks\ holds JB SDK libraries downloaded using the plugin
About should show JRE: 1.8.0_152-release-b941 amd64 JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Find Action > Switch IDE Boot SDK to switch between installed runtimes. Always use the JetBrains version.
38.2.5. Deployment
- https://www.jetbrains.com/help/phpstorm/tutorial-deployment-in-product.html
- Tools > Deployment > Configuration
- For local deployment, use
In placetype, and Web server root URL is the file path to the project root directory e.g.file:///E:/li/cssSandbox/, Mappings > Web path on server is/ Rsync setting
- Default for MacOS
- Rsync executable path
- rsync
- Rsync options
-zar- Shell executable path
- ssh
- Rsync using a non-root but sudoer SSH credentials
- (no term)
- See 15.10.23.4
- Rsync executable path
- rsync
- (no term)
Rsync options
-zar --no-p --no-g --no-o --rsync-path=sudo rsync
- Shell executable path
- ssh
- For local deployment, use
⌥ ⇧ ⌘ x
38.2.6. Search, Find
38.2.6.1. Find Usages ⌥ F7
- When a variable is a whole array, Find Usages works perfectly. If the variable is
$a['abc'], Find Usages won't show functions that change$a['abc'](not the whole array$a) asValue writebut show as underValue read.$a['abc'][] = 'etc';is always shown asValue write
38.2.6.2. Find file
- Search Everywhere
- double tap
S - Recent Files
C-eor⌘ e- Switcher
⇧ tab- d
- switch to database view
- Navigate > File (Go to file)
C-S-r- Find directory
name/orname\
- (no term)
- Find file
- CamelHumps
sapmatchesStrangeAnimalPage.html- snake_case
s a pmatchesstrange_animal_page.html- Files in nested directories
a/a/s a pmatchescalendar\Animals\Animaux\StrangeAnimalPage.htmlandstrange_animal_page.htmlin the same folder- (no term)
- e.g. for Drupal,
s/a/m/custom/*.mdmatchessites/all/modules/custom/*.mdfiles
38.2.6.3. Regex Find
- Find functions with names that contain
_node_function[\s+].*_node
- Functions with names that end with
_node_function[\s+].*_node\(
- Function calls with specific values for optional arguments
- Not 100% correct but close enough
- e.g. PHP function
currency_format( $val, $digits=2, $bracket=0, $omitzero=0 ) - Want to see all function calls with $bracket = 1
currency_format\(.*,\s*2\s*,\s*1\s*currency_format\([^\)]*,[^\)]*,\s*2\s*
- Multiple lines
:PROPERTIES:\n:ID:.*\n:END:\n
38.2.6.4. Search Structurally
38.2.7. Language Specifics
38.2.7.1. HTML: Show Applied Styles
Right click on element and select Show Applied Sytles for Tag
38.2.7.2. CSS: vendor prefix
e.g. all vendor prefixes for box-sizing. Type -box-sizing and hit tab
38.2.7.3. JavaScript version
- Settings > Languages & Frameworks > JavaScript > JavaScript language version to ECMAScript 6
- For Scratch, right click on the file in Edit window and change language to ES6
38.2.8. View
38.2.8.1. Appearance View > Appearance
- View modes
- e.g. Zen mode
C-S-F12View > Appearance > Enter Zen Mode- Hide all active tool windows and leave the Editor Tabs open. Can toggle
- (no term)
- Tool Window Bars
- appears on left of Editor Tabs
- (no term)
- Status Bar
- Status Bar Widgets
- select all items to display
- (no term)
- Navigation Bar
M-Home - (no term)
- Details in Tree View
- I choose off
- In Project, show file dates and size
- In Database, show description for databases and tables
38.2.8.2. Quick Switch Scheme, Switch Theme C-`, View > Quick Switch Schme
- Editor Color Scheme
- Code Style Scheme
- default
- for all projects (default)
- Project
- for current project
- I choose Eclipse to be my default keymap
- presentation mode, fullscreen mode, zen mode
- Theme
- Material Theme presets
38.2.8.3. Parameter Info
- Inside the function parameter
abc(placeCursorHere) - 1.1
38.2.8.4. Quick Definition C-S-i
38.2.10. Navigate, Display, Select inside Editor
38.2.10.1. Edit > Paste > Paste from history (access multiple clipboards) :: C-S-v ⌘ ⇧ v
38.2.10.2. Edit > Find
- Replace
⌘ r
- Search Structurally
- https://www.jetbrains.com/help/phpstorm/2022.2/structural-search-and-replace.html
- Saved templates are stored per project basis not throughout the IDE
- Choose language first!
- Save template as Inspection
- Give a name and description
- Go to Inspections in Settings to modify the severity level
- Static method calls of a class
<searchConfiguration name="Utils::debugDump" text="$a$::debugDump($c$)" recursive="true" type="PHP" search_injected="false"> <constraint name="__context__" within="" contains="" /> <constraint name="a" nameOfExprType="BrokerToMAUtils" within="" contains="" /> <constraint name="c" minCount="3" maxCount="2147483647" within="" contains="" /> </searchConfiguration>
- Static method calls of a class using regex
- The 3rd parameter not starting with
0orwhitespace- When
^is used in a regex, better to use$to match the whole 3rd parameter
- When
<searchConfiguration name="Utils::debugDump" text="$a$::debugDump($b$, $c$, $d$, $e$)" recursive="true" type="PHP" search_injected="false"> <constraint name="__context__" within="" contains="" /> <constraint name="a" nameOfExprType="BrokerToMAUtils" within="" contains="" /> <constraint name="c" within="" contains="" /> <constraint name="b" within="" contains="" /> <constraint name="d" regexp="^[^0\s].*$" within="" contains="" /> <constraint name="e" minCount="0" maxCount="2147483647" within="" contains="" /> </searchConfiguration>
- The 3rd parameter not starting with
- Static method calls of all child classes of a specific parent class
First, create a search template
BasicEnumChildClassesthat finds all child classes of a parent class calledBasicEnumabstract class $BasicEnumChildClass$ extends $b$ {}<searchConfiguration name="BasicEnumChildClasses" text="abstract class $BasicEnumChildClass$ extends $b$ {}" recursive="true" type="PHP" case_sensitive="true"> <constraint name="__context__" within="" contains="" /> <constraint name="b" regexp="BasicEnum" within="" contains="" /> <constraint name="BasicEnumChildClass" target="true" within="" contains="" /> </searchConfiguration>
Create another search template
BasicEnumChildClass Static Method Callswhich reuses$BasicEnumChildClass$from the previous template$BasicEnumChildClass$::$b$()<searchConfiguration name="BasicEnumChildClass Static Method Calls" text="$BasicEnumChildClass$::$b$()" recursive="true" type="PHP" case_sensitive="true"> <constraint name="__context__" within="" contains="" /> <constraint name="b" within="" contains="" /> <constraint name="BasicEnumChildClass" reference="BasicEnumChildClasses" within="" contains="" /> </searchConfiguration>
- Function Calls
sql_fetch_assoc($a$($p$), $pp$)sql_fetch_assoc followed by function calls of sql_query or sql_prep
<searchConfiguration name="sql_fetch_assoc($a$($p$), $pp$)" uuid="d8bdd089-5864-3ad5-8649-fad285ed10f8" text="sql_fetch_assoc($a$($p$), $pp$)" recursive="true" type="PHP" case_sensitive="true"> <constraint name="__context__" within="" contains="" /> <constraint name="a" regexp="sql_prep|sql_query" within="" contains="" /> <constraint name="p" minCount="0" maxCount="2147483647" within="" contains="" /> <constraint name="pp" minCount="0" maxCount="2147483647" within="" contains="" /> </searchConfiguration>
- My Search Templates
38.2.10.3. Edit > Find Usages
- Find Usages
C-gor⌥ F7- If a JavaScript method is called inside a PHP Heredoc, you'll need to open the Edit Fragment, then you can Find Usages at the place where the JS method is defined
- You can still click on the called method insdie PHP Heredoc to locate the JS method declaration place
- TS: Control flow is too big to analyze
- When a file has too many lines inside a block scope e.g.
switchwith many lines - or a file has a lot of function/method calls inside a block scope
- The inspection (e.g. Find Usages, Autocompletion) is turned off for that file
- When a file has too many lines inside a block scope e.g.
- If a JavaScript method is called inside a PHP Heredoc, you'll need to open the Edit Fragment, then you can Find Usages at the place where the JS method is defined
- Next highlighted usage
⌃ ⌥ Down / Up- Just need to hover on the variable and hit to see the next usage
- Highlight usages in File
⇧ ⌘ F7- Find Usages in File
- Edit > Find Usages > Find Usages in File
⌥ ⌘ F7 - (no term)
- See PHPStorm Structural Search
- (no term)
- Find Usages Settings
- Scope
- default Project Files and Libraries
- (no term)
- General
- Search for text occurrences
- default unchecked
- Skip results tab with one usage
- default unchecked
38.2.10.4. Show Whitespaces - Show unprintable characters (TAB)
Find Action > search Show Whitespaces
38.2.10.5. Multiple cursors, move cursor, selection, navigate cursor
- Create a cursor at caret
M-clickor- Hit
Ctrl/⌥twice and holdCtrl/⌥, then press an arrow key to create a cursor Clone Caret BeloworClone Caret AbovewhenC-S-a(no shortcuts defined in Eclipse)
- Hit
- (no term)
- Select and make new cursors
- Add selection for next occurance
M-y- Select all occurances
C-M-y
- Reduce to only one cursor
Esc
38.2.10.6. Extend selection :: S-M-Up S-M-Down ⌥ Up
38.2.10.7. Move Caret to Line End :: End ⌘ Right ⌃ e
38.2.10.8. Move Caret to Line Start :: Home ⌘ Left ⌃ a
38.2.10.9. Move Caret to Code Block Start or end (open bracket, starting bracket) :: C-[, C-], ⌥ ⌘ [
- Repeat to keep going up to parent scope code blocks
38.2.10.10. Move Caret to Matching Brace :: C-S-p ⌃M
- First hit goes to the start brace, hit again to go to the end brace
([<"'
38.2.10.11. Move Caret to Text Start (top of file) :: ⌘ Home
38.2.10.12. Navigate > Class C-S-t, File C-S-r, Symbol C-M-S-n
38.2.10.13. Navigate > Jump to navigation bar ⌘ up
38.2.10.14. Navigate > File Structure (popup window) :: C-F3 ⌘ F12
38.2.10.15. Navigate > Type Hierarchy F4, Method Hierarchy, Call Hierarchy
- Type
- Class inheritance hierarchy
- Method
- Hierarchy of a method of a class
- Defined in this class, superclasses? do subclasses also define this method?
- (no term)
- Call
- Sometimes Expand All to get callers and callees will end up in infinite loop
- Caller Hierarchy
- Which functions call this method/class? And which functions call those functions?
- Callee Hierarchy
- Which functions get called by this method/class? And which functions also call those functions?
38.2.10.16. Navigate > Implementation(s) C-t ⌥ ⌘ B
38.2.10.17. Navigate > Super Method (opposite of Navigagte > Implementaion(s)) :: ⌘ U
38.2.10.18. Navigate > Declaration and Usages F3 or C-leftClick or ⌘ + b
38.2.10.19. Navigate > Select in (select opened file in other window) ⌥ + F1
38.2.10.20. Navigate > Column (Go to line) ⌘ + L
38.2.10.21. Navigate > Back (move to previous place on editor) :: ⌘ [
38.2.10.22. Navigate > Last edit location :: ⇧ + ⌘ + Backspace
38.2.10.23. Navigate > Next Highlighted Error (next error in an Editor tab) :: S-F1 Mac: F2 ⇧ F2
38.2.10.24. Navigate > Navigate in File > Next Change, Previous Change (Git) :: ⇧ ⌃ ⌥ Up/Down
38.2.10.25. Navigate > Navigate in File > Previous method / Next method :: C-S-Up, C-S-Down, ⇧ ⌃ Up/Down
- Move to previous sibling element in HTML. If there's none, then move to the start of the parent element
38.2.10.26. View > Recent Locations C-S-e M-left or right
38.2.10.27. View > Parameter Info (function parameters) :: ⌘ P, ~~
38.2.10.28. Error Description :: ⌘ F1
38.2.10.29. Scroll to Centre ⌃ l
38.2.10.30. Custom code folding regions
- Select the block of code you want to fold
C-M-torsurround with- Choose either NetBeans style or VisualStudio style. Don't mix 2 styles
38.2.10.31. Window > Editor Tabs - Split and multiple Editor Tabs
38.2.10.32. Window > Active Tool Window > Maximize Tool Window ⇧ ⌘ '
38.2.10.33. Window > Active Tool Window > Hide all windows :: ⇧ + ⌘ + F12
- required
- Windows > Store Current Layout as Default
38.2.10.34. Window > Next project window ⌘ ⌥ `
38.2.10.35. Open source in new window (open current file in new window) :: S-F4 ⇧ F4
38.2.10.36. Close tab C-F4
38.2.10.37. Show context menu ⌥ + Enter
38.2.10.38. Quick Definition :: C-S-i ⌥ Spacebar
38.2.10.39. Quick Documentation (has browser support for CSS) :: C-S-Space F1
38.2.10.40. Goto by Reference Actions - File Path ⌥ ⌘ F12
38.2.10.41. Goto by Reference Actions - Jump to Navigation Bar :: ⌘ Up
38.2.10.42. VCS Operations ⌃ + v
38.2.11. Difference Viewer
⌃ + ⌘ + D- Toggle viewer control options e.g. Ignore options, Highlight, collapse unchanged parts
⌃ + ⇧ + Tab- Switch between left and right pane
F7or⇧+F7- Next / previous difference
⇧ + ⌘ + [- Compare next / previous file
⌘ + Down- Jump to source file
- (no term)
- Highlight
- Highlight split changes
38.2.12. Project View / Project Tree ⌘ 1
38.2.12.1. 3 dots > Always Select Opened File
38.2.12.2. Collapse All ⌘ -
38.2.12.3. Rename Project
- From Project View, select the root folder of the current project, and go to File > Rename project
38.2.13. Git ⌘ 9
- Branches > right click a branch e.g. master and the current branch is release
- Show Diff with Working Tree
- Will show file difference.
Changestab will open on sidebar
- Will show file difference.
- or the current branch
- An editor tab will open to show commit difference
- Show Diff with Working Tree
- see 38.2.16.6
38.2.14. Edits inside Editor
38.2.14.1. Line Comment C-/, Block Comment C-S-/
38.2.14.2. To Spaces / To Tabs (convert to Tab/Spaces for indentation) :: Find Action > To Tab
38.2.14.3. Surround with M-S-z or C-M-t or ⌥ ⌘ t
- Surround with Live Template
- Select a text abc and wrap it into a function call
- Settings > Live Templates > under user, +, Live Template >
- Abbreviation
lifc- Description
- Surround with function call
- Template text
$END$($SELECTION$)- Context
- Everywhere
- Settings > Live Templates > under user, +, Live Template >
- Select the text, Code > Surround with, or
M-S-z
- Select a text abc and wrap it into a function call
- Surround with Emmet phpstorm:emmet:wrap
38.2.14.4. Code > Insert Live Template C-M-S-j
- See 38.2.16.1.4
- See 38.2.14.3.1
- HTML
- PHP
- pubf
- public function
- pubsf
- public static function
- prif
- private function
- prisf
- private static function
- prof
- protected function
- prosf
- protected static function
- thr
- throw new
- inco
- include once
- forek
- foreach(iterable_expr as $key => $value)
- eco
- echo
38.2.14.5. Postfix Template 38.2.16.1.4
38.2.14.6. Code > Generate… M-Insert
- Generate PHPDoc Blocks
- A manual way is go to the previous line and type
/** Enter, PHPDoc will be generated
- A manual way is go to the previous line and type
- Constructor, Getter, Setter
- Override methods
- Don't have an option to override fields..
- Implement methods
- Magic methods
38.2.14.7. Code > Folding Fold Selection / Remove region C-.
- Fold or unfold the parent of current code
- Apply on integer in query to show the integer with thousand separator
_e.g.1000shows1_000
38.2.14.8. Code > Move Line Down / Up ⇧ ⌥ Down
38.2.14.9. Code > Auto-Indent Lines (auto indent) :: ⌃ ⌥ i
38.2.14.10. Code > Analyze Code > Configure Current File Analysis :: ⌥ ⇧ ⌘ h
Turn off error highlighting, turn off highlight
38.2.14.11. Code > Code Completion
- Basic
⌃ Space- Complete names of classes, methods and keywords within the visibility scope
- Include Live templates
- Type-matching
⌃ ⇧ Space- Complete Current Statement
⇧ ⌘ Enter- Cyclic Expand Word
⌥ /⇧ ⌥ /
38.2.14.12. Show Context Actions (show intention actions) :: M-Enter ⌥ Enter
- At code line, press
M-enter - May provide some refactoring options
- Convert to sprintf
- Convert to string interpolation
- if a Class function can be made to a static function
- move cursor to arguments then hit the shortcut
- Language Injection
- For example, insert HTML code to a PHP variable. We need syntax highlighting of that HTML code inside the PHP file
- https://www.jetbrains.com/help/phpstorm/using-language-injections.html#use-lang-annotation
- Use PHP nowdoc or heredoc
- Inside HereDoc
Alt+Enterand chooseEdit JavaScript Segment
- Inside HereDoc
- Inline
- Place cursor in front of a string,
M-Enter>Inject Language using PHPDoc
- Place cursor in front of a string,
$a = /** @lang JavaScript */ "console.log('hello')";- Add a comment line above the variable assignment line
// language=CSS
- SQL, RegExp, XML, HTML, text
- Place cursor in front of a string,
M-Enter>Inject language or referrenceto temporarily set the language
- Place cursor in front of a string,
- SQL Dialect
- Settings > Editor > Language Injections
Manually find an entry named
php: <<< SQL. Duplicate it and changeSQLtoMYSQL, now heredoc will use the MySQL dialect$s=<<<MYSQL SELECT * FROM `table`; MYSQL;
38.2.14.13. Fix Doc Comment Find Action > fix doc comment
- This is the only way to insert or fix JSDoc
38.2.14.14. Emmet
- How it works
snippets.json store something like "m": "margin:|;"
Settings > Emmet > HTML, CSS, JSX Settings > Keymap > search emmet
- Surround with, wrap each line with a tag phpstorm:emmet:wrap
Refer to phpstorm:surround with
<!-- * Unordered item 1 * Unordered item 2 * Unordered item 3 Select 3 lines and type (abbreviation) ul>li* --> <ul> <li>* Unordered item 1</li> <li>* Unordered item 2</li> <li>* Unordered item 3</li> </ul> <!-- Use filter t (trim) to remove list markers - ul>li*|t - ul.nav>li.nav-item$*>a|t -->
- Number $
- Repeat input * as $#
- Filters using Pipe(s)
- HTML
- CSS
38.2.14.15. Start New Line Before Current ⌥ ⌘ Enter
38.2.14.16. Edit > Join lines C-S-j ⌃ ⇧ j
38.2.14.17. Edit > Toggle Case ⇧ ⌘ u
38.2.14.18. Window > Editor Tabs
- Maximize Editor / Normalize Splits
⌥ Tabor⇧ ⌥ Tab
38.2.14.19. Toggle Rendered View - rendered PHPDoc ⇧ ⌥ q
38.2.15. Settings > Directories
38.2.15.1. Exclude Directories
- Settings > Directories > Select and Exclude
- Settings > Deployment > Excluded Paths
- https://www.jetbrains.com/help/phpstorm/configuring-folders-within-a-content-root.html
- Webpack config file might specify output dir and this dir is excluded and cannot be un-exclude using Settings > Directories > Exclude. Instead, remove the config file on phpStorm Settings > Languages & Frameworks > JavaScript > Webpack
38.2.15.2. Resource Root
- This should be set to where the final CSS files are placed
- e.g. assets/css/style.css has
background-image: url(../abc.jpg)- Then the Resource Root should be set to assets/css
38.2.16. Settings > Editor
38.2.16.1. General
- General
- On Save
- Turn off Remove trailing spaces
- Soft Wraps
- Only show soft-wrap indicators for the current line
- default on, turn it off
- On Save
- Code Completion
- Parameter Info
- Show full method signatures
- default off
- See 1
- database auto completion
- Qualify object with
- database auto completion
- Parameter Info
- Code Folding
- Fold by default
- XML
- XML entities
- disable to always expand HTML entities e.g. show ` ` instead of a visual space
- XML
- Fold by default
- Postfix Completion
- PHP
array_walk($EXPR$, function ($v, $k) use (&$r) { $END$ }, $r = '');
array_reduce($EXPR$, function ($carry, $item) use (&$externalParam) { $carry[] = $item; return $carry; }, array());
array_filter($EXPR$, function ($v, $k) use (&$externalParam) { return $v > 0; }, ARRAY_FILTER_USE_BOTH);
// arrow (function () { $EXPR$ })()
array_map(function ($v) use (&$externalParam) { return $v; }, $EXPR$);
$END$($EXPR$ ?? null)
- PHP
- Smart Keys
- Reformat on paste
- indent each line (default)
- Use "CamelHumps" words
- selection expansion and move caret to word using CamelHumps or underscore. Word separator / word boundary is underscore.
- (no term)
- PHP
- Enable smart function parameters completion
- off
38.2.16.2. Color Scheme
- General
- Code
- Identifier under caret
- Identifier under caret (write)
- Code
- Language Defaults
- Semantic highlighting
- enabled
38.2.16.3. Code Style
- For each language
- Use Project scheme
- e.g.
Set fromWordPress Change Tab size and Indent from 4 to 2
- e.g.
- Set a default
Code Style Schemee.g. Project sheme and Reformat code will use that scheme- For IDE scheme, stored at C:\Users\Li\AppData\Roaming\JetBrains\PhpStorm2020.1\codestyles or can export scheme
- Use Project scheme
- PHP
- PSR-1 < PSR-2 < PSR-12
- Basic Coding standards https://www.php-fig.org/psr/psr-12/
- PSR-0 < PSR-4
- Namespaces, class names and file paths
- (no term)
- Use PSR-12 and do some manual changes
- Code Generation
- Comment Code
- disable Line comment at first column, and enable Add a space at comment start
- Wrapping and Braces
- 'if()' statement
- 'else' on new line
- Force braces
- try statement
- catch on new line
- finally on new line
- 'if()' statement
- Code Generation
- JavaScript
- Punctuation
- Use semicolon to terminate statements always
- Punctuation
- HTML
- https://google.github.io/styleguide/htmlcssguide.html
- don't have an xml for phpStorm.. need to do it manually
- (no term)
- Other
- Do not indent children of
- add script
38.2.16.4. Inspections
No inspection
/** @noinspection PhpReturnValueOfMethodIsNeverUsedInspection */ // noinspection PhpReturnValueOfMethodIsNeverUsedInspection
- PHP
- https://gist.github.com/jorgecc/c17e4f95d50fe14c412333129502ba30
- https://gist.github.com/gskema/66f82792f2221771ea7407ff5245b35a
- PhpReturnValueOfMethodIsNeverUsedInspection
- PHP
- PHP Mess Detector (PHPMD)
- PHP > Quality Tools > Mess Detector
- Static code analyzer
- If a remote interpreter is used, the default
System PHPshould not be used. Add one- The
PHP Mess Detector pathshould be the path in the Docker container (remote) - Enable all rules under
Options
- The
To get the executable, you will create a new folder with one composer.json file
{ "require-dev": { "phpmd/phpmd" : "@stable" } }cd path/to/folder && composer install. The executable ispath/to/folder/vendor/bin/phpmd- https://phpmd.org/download/
path/to/folder/vendor/phpmd/phpmd/src/main/resources/rulesets- cleancode.xml
- codesize.xml
- controversial.xml
- design.xml
- naming.xml
- unusedcode.xml
- my-custom-ruleset.xml
path/to/folder/vendor/bin/phpmd path/to/my.php my-custom-ruleset
- Editor > Inspections > PHP > Quality tools > PHP Mess Detector validation
- Customize ruleset
<?xml version="1.0"?> <ruleset name="My first PHPMD rule set" xmlns="http://pmd.sf.net/ruleset/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pmd.sf.net/ruleset/1.0.0 http://pmd.sf.net/ruleset_xml_schema.xsd" xsi:noNamespaceSchemaLocation=" http://pmd.sf.net/ruleset_xml_schema.xsd"> <description> My custom rule set that checks my code... </description> <!-- Import the entire unused code rule set --> <rule ref="rulesets/unusedcode.xml" /> <rule ref="rulesets/codesize.xml"> <exclude name="CyclomaticComplexity" /> </rule> <!-- Import the entire cyclomatic complexity rule and customize the rule configuration. --> <rule ref="rulesets/codesize.xml/CyclomaticComplexity"> <priority>1</priority> <properties> <property name="reportLevel" value="1" /> </properties> </rule> <rule ref="rulesets/naming.xml" /> <rule ref="rulesets/cleancode.xml" /> <rule ref="rulesets/controversial.xml" /> <rule ref="rulesets/design.xml" /> </ruleset>
- PHP > Quality Tools > Mess Detector
- General
- PHP
- Refactoring opportunities
- Complex class should be refactored
- Complex function should be refactored
- Cyclomatic complexity
- default 15
- Function has too high cyclomatic complexity
- Cyclomatic complexity
- default 20
- LCOM metric
- Lack of Cohesion of Methods
- Refactoring opportunities
- SQL
- Unresolved reference
- Enable
Suppress for quoted identifiers which may be treated as strings- But double quotes in
INSERT ... VALUESstill throw errors
- But double quotes in
- Enable
- Unresolved reference
- JavaScript and TypeScript
- Function metrics
- Overly complex function
- Cyclomatic complexity
- default 10
- Overly complex function
- Function metrics
38.2.16.5. File and Code Templates
38.2.16.6. File Types
- Ignore Files and Folders
- Remove
.gitto show.gitfolder under Project View- Also consider exclude this folder from index
- In Project View, select the .git folder and Search Action
Excluded. Then Settings > Directories > shows the .git folder is excluded
- Remove
38.2.16.7. Inlay Hints
38.2.16.8. Duplicates
- PHP
- Detect duplicates with different: (ignore names and values)
- Variable or identifier names
- enabled
- Function or field names
- disabled
- Constant values
- disabled
- (no term)
- See 1
- Detect duplicates with different: (ignore names and values)
38.2.16.9. TODO
- Default
\b(todo|fixme)\b.*, show comments with those as prefix in window TODOM-6
38.2.17. Settings > Tools > Database
38.2.17.1. Data Editor and Viewer
- Data Presentation
- Automatically transpose tables
- Always
- Data Sorting
- Sort tables by numeric primary key
- Descending
38.2.18. Settings > Advanced Settings
- Editor Tabs
- Equalize proportions in nested splits
38.2.19. .editorconfig
root = true [*] end_of_line = lf [*.js] indent_size = 2
38.2.20. jsconfig.json
38.2.21. Refactor
- Refactor This..
⌃ t- Rename
⇧ F6- Change Signature (change method/parameter name, parameter order, return type)
⌘ F6- Inline… (inline variable)
⌥ ⌘ n- Remove the variable assignment line and move the value into the place where the variable is called
- Opposite of Extract Variable
- (no term)
- Make Static
- (no term)
- Extract/introduce
- Variable
⌥ ⌘ v- Constant
⌥ ⌘ c- Method
⌥ ⌘ m- Field (extract to a class property)
⌥ ⌘ f- Parameter (extract to a function parameter)
⌥ ⌘ p- (no term)
- Interface
- Place caret at a class name, a new file of interface is created, and the current file will implement the newly created interface
- (no term)
- Extract Class
- Place caret at a class method, select additional methods, extract them into another class
- Pull members up / Push members down
- move methods to parent/child class
38.2.21.1. Move Class F6
- https://www.jetbrains.com/help/phpstorm/keeping-namespaces-in-compliance-with-psr0-and-psr4.html
- PhpStorm can auto detect namespace roots if the project has
composer.jsonwithautoloadsection - May also configure namespace roots manually
- Settings > Directories
- On the left, pick a root and mark it as
Sources - On the right, click the pencil icon and specify the
Package prefix
- On the left, pick a root and mark it as
- Settings > Directories
- PhpStorm can auto detect namespace roots if the project has
38.2.22. Help
38.2.22.1. Find Action ⇧ + ⌘ + a
- On Mac, there's a shortcut key binding conflict. Turn it off or use Search Everywhere on PHPStorm instead
S+S
38.2.22.2. Change Memory Setting :: 4096MB, default 2048MB
38.2.22.3. Delete Leftover IDE Directories
38.2.23. Database
38.2.23.1. Setup for a project
- Settings > Languages & Frameworks > SQL Dialects
- Project SQL Dialect
- Add specific files to use a different dialect from the project SQL dialect
- Project SQL Dialect
- See 2
38.2.23.2. Go to DLL :: ⌘ b
38.2.23.3. Select in Database explorer :: ⌥ ⇧ b
38.2.23.4. Full text search on a table :: ⌥ ⇧ ⌘ f
38.2.23.5. Data Editor (table view)
- Edit Filter Criteria (go to where filter)
⌥ ⇧ ⌘ f- Columns List (popup)
⌘ F12- Go To Row…
⌘ l- Go to row #1
1- Go to column abc at row #1
abc:1
38.2.23.6. Console
When there're multiple statements (multiple ;), C-Enter (or to Execute from Menu) prompts a list of statements to run:
- Smallest statement
- The smallest of the possible statements is executed. For example, when the cursor is inside a subquery, the subquery is executed.
- Largest statement
- The largest possible statement is executed.
- e.g., when the cursor is inside a subquery, an outer statement is executed
- Largest statement or batch
- For Transact-SQL (SQL Server and Sybase), the current batch of statements is executed. For all other dialects - the same as the previous option.
- Whole script. All the statements are executed. (This item always appears and always appear last)
- You can also
C-ato select all statements andC-Enterto run all of them
38.2.23.7. Return all query results
- Click on the Wrench icon on the toolbar of the Database Console tool window
- Switch to Database | Data Views page, specify 0 in the Result set page size field, and click OK
- Default is 500
- That way you can export all rows to csv file
38.2.23.8. Icons
- https://www.jetbrains.com/help/phpstorm/database-tool-window.html
- Default description appears next to each column shows
bigint(20) unsigned = 0bigint(20) unsigned (auto increment)- But it doesn't show whether it is
NOT NULL, indexed or primary/foreign key. Need to rely on icons - a circle on the bottom left
- the left bar is highlighted (in blue)
- yellow key
- blue key
38.2.23.9. sql_mode or other session variables are not followed by global variables
- Reason
- JDBC driver might override and set session variables differently than global variables set on MySQL target server
- Solutions
- choose 1
- Change on the Data Source (connection)
- worked for me 20220831
- Options > Startup script
SET SESSION sql_mode='NO_ENGINE_SUBSTITUTION'
- Options > Startup script
- (no term)
- Change on the Drivers (NySQL)
- Advanced > sessionVariables
sql_mode='NO_ENGINE_SUBSTITUTION'
- Advanced > sessionVariables
38.2.24. Plugins
38.2.24.1. Makefile support
38.2.24.2. Makefile Navigator
Turn off the plugin:Makefile support and enable this plugin
38.2.24.3. Themes
- One Dark theme
- There're 2 separate theme settings in PhpStorm:
- Main theme used in UI and Editor: Settings > Appearance & Behavior > Appearance > Theme
- Theme specifically for Editor: Settings > Editor > Color Scheme > Scheme
- If there's any problem, make sure these 2 settings use the same theme
- There're 2 separate theme settings in PhpStorm:
- Cyan Light Theme
- Atom One Theme
- Dark Purple Theme
- Gradianto
- Hiberbee Theme
38.2.24.4. Live Edit
38.2.24.5. JSON Sorter
38.2.24.6. PostCSS
38.2.24.7. Unit File Support (systemd)
38.2.24.8. .ignore
38.2.24.9. Ideolog
Interactive viewer for '.log' files
38.2.24.10. Uninstalled
- Rainbow Brackets
- Atom OneDark Theme (updated 3 years ago)
- Xcode-Dark Theme
- CodeTogether
- Remote pair programming
- TextMate
Install plugin TextMate bundles support Settings > Editor > TextMate Bundles Add a git repo to support Python https://github.com/textmate/python.tmbundle
- JS GraphQL
- Python
- nodejs phpstorm:nodejs
Enable ES6 for Settings > Languages & Frameworks > Javascript > JavaScript language version > ECMAScript 6 Install plugin NodeJS, restart phpStorm after Node and NPM are installed on Windows. Enable
Node.js Core libraryin Settings > Languages & Frameworks > Node.js and NPM Node interpreter looks likeC:\Program Files\nodejs\node.exe - Golang phpstorm:golang
Search for Go, install plugin and restart phpStorm Setup SDK, which is c:\go
- Org4Idea
38.2.24.11. Presentation Assistant
38.2.25. Docker support
In Docker for Windows > Settings > General Choose Expose daemon on tcp://localhost:2375 without TLS
In phpStorm, Settings > Build, Execution, Deployment > Docker, create Docker with a + button.
Default should work, so just OK
API URL: tcp://localhost:2375 Certificates folder: blank Docker Compose executable: docker-compose VirtualBox shared folders: VM Path - /c/Users, Local path - C:\Users
You should see Docker under the bottom bar, select and connect to it. Now you should see the running containers and available images.
38.2.26. Vue.js
JetBrains plugin repository
38.2.27. WordPress Support
Settings > Languages & Frameworks > PHP > Frameworks > WordPress > Enable WordPress integration
- Specify installation path
- May also need to add the installation path to Settings > Languages & Frameworks > PHP > Include Path
38.2.28. Xdebug
38.2.28.1. Basics
- Since XDebug 2.0, protocol DBGp is used
- Set up xdebug:config
- Restart the image:php:7.x-fpm container or apache:restart or nginx:restart
- When one of these happens, XDebug server will try to connect IDE debugger:
- URL
XDEBUG_SESSION_START=session_nameXDEBUG_PROFILE=1
XDEBUG_SESSION_STARTas POST parameter- a cookie named
XDEBUG_SESSION- Install Chrome extension which adds a cookie in order to enable XDebug
export XDEBUG_CONFIG="idekey=session_name remote_host=localhost profiler_enable=1"in XDebug host
- URL
Start Listening for PHP Debug Connectionson phpStorm- IDE's debugger works by listening the remote_port. It works without setting up any CLI interpreter
- Only enable
Use path mappingsand specify the Path Mapping. Leave defaults for Name:_, Host:_, Port:80, Debugger:Xdebug - The added server can be found later under Language & Frameworks > PHP > Servers
38.2.28.2. For lando:xdebug
- https://www.drupaleasy.com/blogs/ultimike/2018/01/setting-xdebug-lando-and-phpstorm
- https://www.youtube.com/watch?v=sHNJxx0L9r0
For per project to include third-party code (e.g. libraries) that is used for completion and reference resolution in functions/methods that use file paths as arguments, for example, require() or include()
Settings > Languages & Frameworks > PHP
- choose the correct php version
- Add CLI interpretor > remote is Docker, for Server, if Docker is not available, hit New > Unix socket
- Image name
- choose the image with the corresponding PHP version e.g.
devwithland/php:7.0-fpmfor Pantheon project - The following fields are by default
- PHP executable: php. Hit Apply
- and then include:
~/.lando/services/config/pantheonor WindowsC:\Users\li\.lando\services\config\pantheon- Or other recipe
C:\Users\li\.lando\services\config\drupal8 - The point is to load
prepend.phpwhich defines global vars related to environments
Start Listening for PHP Debug Connections, then load the local website, accept the Incoming Connection from XDebug- Settings > Languages & Frameworks > PHP > Servers will show the newly accepted connection as a server. Make sure the following is correct
- host: appserver, port: 80, debugger: XDebug
- project files:
your-local-directory-of-the-project, absolute path on server:/app(very important) - include path:
C:\Users\li\.lando\services\config\pantheon, absolute path on server:/srv/includes
38.2.28.3. For Vagrant
- Settings > Languages & Frameworks > PHP
- Choose correct PHP version
- Install PHP on Vagrant 1.1.1
- Add CLI Interpreter > type is Vagrant, choose the path of the Vagrantfile. Everything should be auto set up:
- Vagrant Instance Folder: C:\c
- Varant Host URL: ssh://vagrant@127.0.0.1:2222
- /usr/bin/php7.3
- Configuration file: /etc/php/7.3/cli/php.ini
- Settings > Languages & Frameworks > PHP > Quality Tools
- Refer to #php:composer:php_codesniffer
- Then Editor > Inspections > PHP > Quality Tools > enable
PHP_CodeSniffer validation, and choose one coding standard
- Run Code > Inspect Code > select scope, double check Quality Tools settings, run
PHP CODE BEAUTIFIER AND FIXERauto fix is available to some problems listed under PHP > Quality Tools > PHP_CodeSniffer validation
38.2.28.4. For Docker > Vagrant > Windows
- Do the same as vscode:remote:ssh:proxy
- Settings > Language & Frameworks > PHP
- Choose correct PHP version
- New CLI Interpreter, type
SSH Credentials- Host, Port, User name have to be filled. Same as Windows local
~/.ssh/configforremotecontainer - Auth type: OpenSSH config and authentication agent
- Select the CLI php. Find it in Docker container by
which php
- Host, Port, User name have to be filled. Same as Windows local
<Project root> -> /var/www/html
38.2.28.5. For Docker
- PHP
- PHP Language Level
- 7.3
- (no term)
- CLI Interpreter
- Type
- Docker Compose
- Server
- Docker for Mac
- Configuration files
- the docker-compose.yml file
- Service
- the service/container name. Auto prefilled as docker compose yml is loaded.
- Lifecycle
- Connect to existing container
- PHP executable
- will be auto populated
- Path Mappings
- will be auto populated
38.2.28.6. XDebug config
- https://xdebug.org/docs/all_settings
- Config location image:php:7.x-fpm:ini:extensions
remote_hostcould be- Consider extend timeout
# Type configname = default # boolean xdebug.collect_vars = false # integer xdebug.show_local_vars = 0 # boolean xdebug.remote_enable = false xdebug.remote_enable = 1 # If it's 1, remote_host will be ignored. Xdebug on server will try to connect debugger e.g. IDE using $_SERVER['REMOTE_ADDR'] and xdebug.remote_port # This remote_connect_back only works for HTTP but not CLI request # xdebug.remote_connect_back = 1 # Docker container hosted on Vagrant and IDE is on Windows. Docker container with Xdebug server can connect to Windows using an IP # If Xdebug is on a Docker container hosted directly on Windows, consider use `host.docker.internal` or `localhost` xdebug.remote_host=10.0.2.2 # default. Xdebug server will try to combine remote_host or $_SERVER['REMOTE_ADDR'] and this remote_port to connect the debugger e.g. on IDE over the internet # No port forwarding should be set up as the connection is over the internet # xdebug.remote_port=9000 # req :: default. initiates a session as soon as script is started # jit :: initiate when a session should only be initiated on an error # xdebug.remote_mode = req # Default. Xdebug server does not attempt to try to connect to a debug client e.g. IDE # xdebug.remote_autostart=false # Shows whether or not Xdebug server can connect to a debug client # xdebug.remote_log=/tmp/xdebug.log # xdebug.profiler_enable_trigger = 1
38.2.29. PHPUnit
- php:phpunit
- Set up CLI interpreter as Local or phpstorm:php:cli:ssh proxy
- Languages & Frameworks > PHP > Test Frameworks, add a
PHPUnit LocalorPHPUnit by Remote Interpreter- Select the previously set remote CLI Interpreter, then Path Mappings is auto set
- PHPUnit library: Use Composer autloader
- Path to script:
/var/www/html/vendor/autoload.php- Test Runner
- Default configuration file:
/var/www/html/phpunit.xml
- Run > Edit Configuration, add PHPUnit
- Give a name and only check
Defined in the configuration file
- Give a name and only check
- Now click the run button!
38.2.30. Troubleshoot
38.2.30.1. TS: Reset Index
File | Invalidate Caches and restart IDE
38.2.30.2. TS: Page '…js.map' requested without authorization
Settings -> Build, execution, development -> Debugger - >Allow unsigned request
38.2.30.3. SQLite cannot open local file
- Run as admin to open phpStorm
38.3. Emacs
38.3.1. Starting Guides
38.3.2. Install & Configuration
38.3.2.1. Windows
- Download for Windows https://ftp.gnu.org/pub/gnu/emacs/windows/
- https://ftp.gnu.org/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip
- Previous versions
emacs-26.1-x86_64.zip(with all deps 64-bit)emacs-25.1-i686-w64-mingw32emacs-24.5-bin-i686-mingw32.zip
- Extract it to
C:\emacs, so you will haveC:\emacs\emacs-24.5-bin-i686-mingw32\ - Windows Shortcut to Start
- Make a shortcut of this file
/bin/runemacs.exeand copy to your notes folder sayC:\c\notes\ - Then right click, select Properties of the shortcut and change the
Start IntoC:\c\notes
- Make a shortcut of this file
- Windows Batch File to Start
- Set a HOME user environment variable in Windows as
C:\c\notesand that is~/as the home directory inside Emacs - Include an init file
C:\c\notes\.emacs Home folder in Emacs is
C:\c\notesand from time to time there're auto-save-list (auto saved files records) saved in home folderC:\c\notes\.emacs.dfolder #+NAME C:/c/notes/run.batset HOME=E:\li\notes C:\emacs\emacs-25.1-i686-w64-mingw32\bin\runemacs.exe notes.org
- Set a HOME user environment variable in Windows as
- Extract
emacs-25-i686-deps.zipand copy all files in bin toC:\emacs\emacs-25.1-i686-w64-mingw32\bin
38.3.2.2. Install on Ubuntu 18.04
sudo apt update sudo add-apt-repository ppa:kelleyk/emacs sudo apt install emacs26 -y emacs notes.org
Install linux:font, open Emacs:
Options > Set Default Fontto choose a fontOptions > Save OptionsM-x customize-face RET default RET, click save and apply
- Install for terminal only
sudo apt update && apt upgrade -yand rebootsudo apt install build-essential libncurses-dev- Go to Emacs Download page to get the link. e.g.
wget http://mirrors.syringanetworks.net/gnu/emacs/emacs-26.1.tar.gz tar -xzvf emacs-26.1.tar.gzcd emacs-26.1/./configure --without-x --with-gnutls=nomakeandsudo make install- Run emacs
emacs - Change config
emacs ~/.emacs - Copy https://melpa.org/#/getting-started and change the line from
(proto (if no-ssl "http" "https")))to(proto (if no-ssl "http" "http")))
(require 'package) (let* ((no-ssl (and (memq system-type '(windows-nt ms-dos)) (not (gnutls-available-p)))) (proto (if no-ssl "http" "http"))) (when no-ssl (warn "\ Your version of Emacs does not support SSL connections, which is unsafe because it allows man-in-the-middle attacks. There are two things you can do about this warning: 1. Install an Emacs version that does support SSL and be safe. 2. Remove this warning from your init file so you won't see it again.")) ;; Comment/uncomment these two lines to enable/disable MELPA and MELPA Stable as desired (add-to-list 'package-archives (cons "melpa" (concat proto "://melpa.org/packages/")) t) ;;(add-to-list 'package-archives (cons "melpa-stable" (concat proto "://stable.melpa.org/packages/")) t) (when (< emacs-major-version 24) ;; For important compatibility libraries like cl-lib (add-to-list 'package-archives (cons "gnu" (concat proto "://elpa.gnu.org/packages/"))))) (package-initialize)
emacs notes.organd[M-x] package-refresh [RET][M-x] package-list-packages [RET]
38.3.2.3. Install on Mac Catalina
brew install --cask emacs- macos:full disk access to Emacs">Applications > Emacs
- Troubleshoot
- Emacs 29.4 on Mac
Warning: arch-dependent data dir 'Contents/MacOS/libexec/': No such file or directory# shows the installation path which emacs cd /Applications/Emacs.app/Contents/MacOS cp -a libexec-arm64-11 libexec
38.3.2.4. Init file
~/.emacsor.emacs.el- old location for config file
.emacs.d/init.el- macOS and Windows
.config/emacs/init.el- for Linux
Use run.bat to change the HOME directory
#!/bin/bash HOME=~/li/notes emacs -L . --debug-init notes.org
M-: (load user-init-file) RET
38.3.2.5. Get version :: M-x info M-x org-info
38.3.2.6. Command line options
- https://www.gnu.org/software/emacs/manual/html_node/emacs/Emacs-Invocation.html#Emacs-Invocation
- Action arguments
- https://www.gnu.org/software/emacs/manual/html_node/emacs/Action-Arguments.html
- prepend directory to the variable load-path
- A list of directories to search in order for EmacsLisp libraries that you load
38.3.3. Theme
38.3.3.1. Modus Theme
38.3.4. Keybinding
M-- Alt,
⌥ C-- Ctrl,
⌃ S-- Shift,
⇧ s-- Special
⌘
38.3.5. MELPA packages
- Add http://melpa.org/packages/
- Default is just one package-archives "gnu" at http://elpa.gnu.org/packages/
- After adding, a new folder is created
~/.gnupg
- Run
M-x list-packagesto see all packages including built-in pkgs and package-archivesi- installation
u- unmark
x- perform the i and u installation
U- mark available upgrades.
U xis to upgrade all upgradable packages - (no term)
RETto see more about a pkgq- quit
C-x k- kill current buffer
M-x package-refresh-contents- to refresh the package list
- New install is in
~/.emacs.d/elpa/newpkgname-YYYYMMDD.123- May need to follow instructions on Melpa.org to initialize in
.emacsinit file
- May need to follow instructions on Melpa.org to initialize in
- Sometimes the compilation of a package dependes on the core Emacs. If the core is updated, a package may need to be reinstalled
M-x package-reinstall>php-mode
In
.emacs, runpackage-install-selected-packages. The following message1 Packages are not available (the rest already installed). To find out which packages are not available, paste this in*scratch*buffer and run itC-j:(let* ((not-installed (seq-remove #'package-installed-p package-selected-packages)) (unavailable (seq-filter (lambda (p) (not (assq p package-archive-contents))) not-installed))) unavailable)
38.3.6. use-package
38.3.7. GitHub pkg el-get
It can be used to install/load pkgs from difference sources :: https://mrblog.nl/emacs/config.html
Place packages under load-path C-h v load-path RET
C:\emacs\emacs-24.5-bin-i686-mingw32\share\emacs\VERSION\site-lisp is /usr/local/share/emacs/VERSION/site-lisp for a particular Emacs version
C:\emacs\emacs-24.5-bin-i686-mingw32\share\emacs\site-lisp is /usr/local/share/emacs/VERSION/site-lisp for all Emacs version
*.el (source) *.,elc (byte-compiled) are ELPA package and should be used for emacs 24+
(add-to-list 'load-path "~/.emacs.d/lisp/") (add-to-list 'load-path "c:/c/notes/lisp")
38.3.8. M-x: run a command
customize-variable- see a variable value and may change later
M-x customize-variable [RET] package-archives [RET]q- exit the buffer
- emacs:m-x:delete-trailing-whitespace
- delete current buffer's trailing whitespace
- emacs:m-x:indent-region
- reformat indentation for the current buffer without select regions. Or select region then
C-M-\ - Show init time
- emacs:m-x:emacs-init-time
- (no term)
- Use Customize UI to save variables in
.emacsemacs:m-x:customize-option- Variable will be set under
(custom-set-variablesin.emacs
- Variable will be set under
- visual-line-mode
- toggle
- replace tab with whitespaces
M-x tabifyM-x untabify- Toggle minor mode
M-x debug-mode
38.3.9. Help :: C-h
- minor / major modes being used
C-h m- Exit Help buffer
C-x 1- See all help options
C-h C-horC-h ?- Find shortcut by command
where-is C-h w- Find command by shortcut
describe-key C-h k- Description of a function
C-h f- Describe symbol (variable and function)
C-h o- Get variable value
C-h v- Normal hooks
- hook names end with
-hook - Abnormal hooks
- end in
-functions
38.3.10. Basics
- File
C-x C-s,s-s- save file, save current buffer
C-x C-c- quit Emacs
C-x C-f- open file, create file
- Org Mode full Doc
| Editing | |
|---|---|
| underlined | |
verbatim |
|
verbatim |
|
| Italics | |
| C-c . | Insert timestamp |
| M-Backspace | Kill the previous word |
| C-S-Bksp | Kill whole line |
| <q | BEGIN_QUOTE |
| <H | BEGIN_HTML |
| <c | BEGIN_CENTER |
| <e | BEGIN_EXAMPLE |
| c-c ; | BEGIN_COMMENT |
| Headline | |
| TAB | Local tree fold/unfold |
| C-u TAB | Global tree fold/unfold |
| C-u C-u C-u TAB | Show all |
| Table | emacs:table |
| C-c - | Insert a horizontal line below |
| M-S-<down> | Insert a new row above |
| C-c <RET> | Insert horizontal line and a new row below |
| M-S-<up> | Delete a current row |
| M-S-<right> | Insert new column on the right |
| M-S-<left> | Delete current column |
| C-c <SPC> | Blank current field |
| C-c ` | Edit current field |
| S-<TAB> | Move to previous field |
| M-<up>/<down> | Move a row |
| M-<left>/<right> | Move a column |
| C-c | | After text is selected, convert to table |
38.3.11. Temporarily Change Font Size in Current Buffer
- Increase
C-x C-+- Decrease
C-x C--- Restore the default (global) font size
C-x C-0
38.3.12. Windows
- Remove current window
C-x 0- Remove a help window (delete-other-windows)
C-x 1- Split horizontally
C-x 2- Split vertically
C-x 3- Move to other window
C-x o- (no term)
- Scroll page down (scroll-other-window) and up in other window (scroll-other-window-down)::
C-M-v,S-C-M-v - Switch to the other buffer switch-to-buffer
C-x b
38.3.13. Search and replace
C-s- search by pure text, next search result. If search string contains no uppercase, then it's a case-insensitive search
C-r- previous search
M-C-s- regex search
manage_.+_columns- search
manage_*_columns
M-C-r- regex reverse search
M-s o- list-matching-lines regex and show matching lines
C-g- abort search and go to the original position
C-s C-sorC-r C-r- search again
- Regex escape
[ ] \ + * ? .- Regex case-sensitive
- case-sensitive is auto enabled when text contains uppercase
- (no term)
- Emacs Regex
M-x replace-string- Search and replace from the current cursor to the end of buffer
- https://www.oreilly.com/library/view/learning-gnu-emacs/1565921526/ch04s02.html
To force case-sensitve
- Temporarily change a customizable variable
M-x set-variable RETURN,case-fold-search RETURNandnil RETURN. Default value ist
- Before typing the new value, to review the current value
M-n
38.3.14. Region, Selection, Copy & Paste
C-space- start region, then move cursor
C-x h- copy all in buffer, select all
M-@- mark-word. select next word. Repeat to expand
M-h- mark-paragraph. Repeat to expand
C-M-h- mark-defun. select a function
C-gkeyboard-quitquit selectionC-w- cut
C-k- cut from the cursor to the end of the line (not including the newline character)
M-w- copy without cut (copy to kill-ring, shared with other buffers)
C-y- yank (paste)
C-Backspace,M-Backspace- kill one word backwards at the cursor and to kill-ring
M-d- kill one word forwards at the cursor and to kill-ring
C-S-BackspaceorC-S-Delete- kill current line and add to kill-ring
C-a- move to the beginning of line
C-n- move to next line
C-space- start ring
C-a C-k C-/orC-a C-Space C-e M-w- copy current line without the newline and buffer remains unchanged
C-a C-Space C-n M-w- copy current line including the newline and buffer remains unchanged
- (no term)
- Duplicate the current line
- Method 1
C-S-Delete>C-y>C-y- (no term)
- Method 2
C-=to expand region to select the whole line (keep expanding then several lines can be duplicated)- cut whatever is marked
C-y
- Method 3
C-a>C-k>C-k>C-y>C-y- Easy to remember
- Control + akky
C-/,C-_orC-x u- undo
- To redo after an undo
C-g C-_
38.3.15. Moving and basic editing
C-l- Scroll current line to center
C-x b- switch to another buffer
M-<- Move to beginning of buffer e.g. top of file
M->- Move to end of buffer
C-a- go to beginning of line
C-e- go to end of line
C-n- Go to next line
C-p- Move to previous line
C-f- forward like
<right> C-b- backward like
<left> C-v- Page down
M-v- Page up
M-b- Move backward one word
M-f- Move forward one word
M-g M-g- Go to a line
C-c C-j- org-goto. Search in headlines only
- (no term)
- Go back / go to previous location in the local file
C-u C-Space- Jump to the previous mark and remove it from the mark ring
- same as the above in case
C-Spaceon Windows trigger Input Source change
M-^- join the current line with previous line
38.3.16. Headline
C-u TAB- Global tree fold/unfold
- While
Tabcan show the first level child headings, this will show all the nested sub headings - Show all
- While
S-<TAB>- In table, go to previous cell. In other places, global fold/unfold
C-c C-n- Next heading
C-c C-p- Previous heading Go to parent heading if triggered in the content of the heading
C-c C-f- Next heading same level
C-c C-b- Previous heading same level
C-c C-u- Backward to higher level heading
outline-up-heading M-<RET>orC-<RET>- insert heading
C-<RET>org-insert-heading-respect-contentM-<right>/<left>- Decrease/increase level of a parent item only
M-S-<right>- Increase heading level of an item and its child items
M-S-<left>- Same as above but decrease
C-c *- turn a list into heading/headline,
C-c -to turn a heading into list C-x n s- Work on one section
C-x n w- Back to outline
C-c C-j- jump to different heading.
C-gto exit and return to the current position. up/down to move headline,/sparse-tree search - (no term)
M-xorg-fold-show-branchesor-f-br- fold content and only show headings and subheadings
- (no term)
lili-show-path
- (no term)
M-:(org-get-outline-path t)
38.3.16.1. Custom Variables
org-adapt-indentation- nil. Don't indent body, subheadings and bodies of subheadings
38.3.16.2. Property Drawer
- Set a property and create the property drawer if not exists
C-c C-x p- Set
CUSTOM_IDproperty for a headline CUSTOM_ID- CUSTOM_ID is used for HTML element ID in HTML export
- Set
M-x org-id-get-createorM-x o-get- insert a property drawer with the property
ID
- I tried the following to remove random HTML id's and add anchor for headlines using
CUSTOM_ID, but I failed. It creates performance issue
(require 'org-id) ;; (setq org-id-link-to-org-use-id 'create-if-interactive-and-no-custom-id) ;; WARNING! Overwrite core function. (defun org-id-new (&optional prefix) "Create a new globally unique ID. An ID consists of two parts separated by a colon: - a prefix - a unique part that will be created according to `org-id-method'. PREFIX can specify the prefix, the default is given by the variable `org-id-prefix'. However, if PREFIX is the symbol `none', don't use any prefix even if `org-id-prefix' specifies one. So a typical ID could look like \"Org-4nd91V40HI\"." (let* ((prefix (if (eq prefix 'none) "" (concat (or prefix org-id-prefix) "-"))) unique) (if (equal prefix "-") (setq prefix "")) (cond ((memq org-id-method '(uuidgen uuid)) (setq unique (org-trim (shell-command-to-string org-id-uuid-program))) (unless (org-uuidgen-p unique) (setq unique (org-id-uuid)))) ((eq org-id-method 'org) (let* ((etime (org-reverse-string (org-id-time-to-b36))) (postfix (if org-id-include-domain (progn (require 'message) (concat "@" (message-make-fqdn)))))) (setq unique (concat etime postfix)))) (t (error "Invalid `org-id-method'"))) (concat prefix unique))) (defun eos/org-custom-id-get (&optional pom create prefix) "Get the CUSTOM_ID property of the entry at point-or-marker POM. If POM is nil, refer to the entry at point. If the entry does not have an CUSTOM_ID, the function returns nil. However, when CREATE is non nil, create a CUSTOM_ID if none is present already. PREFIX will be passed through to `org-id-new'. In any case, the CUSTOM_ID of the entry is returned." (interactive) (org-with-point-at pom (let ((id (org-entry-get nil "CUSTOM_ID"))) (cond ((and id (stringp id) (string-match "\\S-" id)) id) (create (setq id (org-id-new (concat prefix "h"))) (org-entry-put pom "CUSTOM_ID" id) (org-id-add-location id (buffer-file-name (buffer-base-buffer))) id))))) (defun eos/org-add-ids-to-headlines-in-file () "Add CUSTOM_ID properties to all headlines in the current file which do not already have one. Only adds ids if the `auto-id' option is set to `t' in the file somewhere. ie, #+OPTIONS: auto-id:t" (interactive) (save-excursion (widen) (goto-char (point-min)) (when (re-search-forward "^#\\+OPTIONS:.*auto-id:t" (point-max) t) (org-map-entries (lambda () (eos/org-custom-id-get (point) 'create))))))
38.3.16.3. Drawer
C-c C-x d- Create a drawer
This is inside the drawer
38.3.17. Code Block
- Create a code block
<eandTab- Since Org mode 9.2
- use
C-c C-,
- Inside a code block
C-c + 'to go to a new major-mode edit buffer to edit the code,C-x C-ssaves the buffer and updates the contents of the Org buffer.C-c + 'again to exit- (no term)
Inline code block
src_<language>[<header arguments>]{<body>} src_sh{echo "hello"} src_html[:exports code]{<el></el>} src_haskell[:exports both]{fac 5}- All languages
- html xml css sass js makefile java sh sql sqlite ruby python perl C C++ matlab org dns
- (no term)
- Find modes on GitHub
-modeto*-mode.elfiles - (no term)
- When in
*-modeedit buffer:
38.3.18. Export to HTML C-c C-e h h
- Add markdown and Confluence export format
M-x customize-option>org-export-backends- Arrow down to md and Confluence and
Enter - Arrow back to the top to
Apply and SavethenEnter,qto quit
- Arrow down to md and Confluence and
Include that in the org file that you want to export.
#+SETUPFILE: my-setup.setup
- https://orgmode.org/manual/Export-Settings.html
- Include export keywords
#+OPTIONS: num:nil- overrides global variable
org-export-with-section-numbers- t
- default
- n (an integer)
- numbering will only happen for headlines whose relative level is higher or equal to n
38.3.19. Symbol and Escape emacs:escape
38.3.20. Superscript and subscript
To toggle display: C-c C-x \
x_{sub script}
y^{super script}
38.3.21. TODO
- Only applied to headlines not list items
- Cycle through TODO, DONE and empty
M-S-return,M-x org-insert-todo-headingC-c C-c- toggle checkbox state
38.3.22. Sparse Tree, Search
C-c / r- Search regex,
C-c C-cto quit M-g M-n- Next match
M-g M-p- Previous match
38.3.23. Link
- Here's description of the link
- add or modify link
- Go back to previous position (mark ring)
- Update radio targets
- open a link, go to a link at cursor
Link is defined in this order [[#my-custom-id]] - the entry with `CUSTOM_ID` property set to `my-custom-id`. Don't use because of performance issue [[My Target]] - By defalt, it leads to a text search. `My Target` is a headline with the exact text My Target - Added to Mark Ring go back by using C-c & - Won't work if the headline has dedicated target e.g. <<My Target>> - Works even inside code block! [[*Some section][Link Description]] - same as [[My Target]] but you can use M-Tab to autocomplete headline - does not work on Windows because keyboard shortcut conflict [[internal target]] :: dedicated target - <<internal target>> - Number in front of the internal target will be shown in the places that refer to this target - NAME keyword that is put before the element it refers to [[Table One]] - CAPTION keyword is mandatory in order to get proper numbering #+NAME: Table One | a | table | |----+------------| | of | four cells | |----+------------|
<<<Radio target>>> This causes each occurence of 'Radio target' in normal text to become activated as a link Emacs only update radio targets when the file is first loaded. To manual update, C-c C-c
38.3.24. SSH remote edit files emacs:ssh:tramp
- Basics
C-x C-fand type/ssh:hostsetinsshconfig:/pathtofileordir
- Windows
- Need to use Putty (GUI to set a session) and Plink (CLI SSH client used by Emacs). Use this command
/plinkx:sessionsetinputty:/pathtofileordir - Need to start up Emacs using PowerShell
- Add the folder of
plink.exeto env. varPATH. Check$env:PATHin PowerShell
- Need to use Putty (GUI to set a session) and Plink (CLI SSH client used by Emacs). Use this command
38.3.25. Superscript and Subscript
By default Sometext_subscript and Sometext^superscript shows sub/super scripts when .org is exported.
Have this setting: Org > Customize > Browse Org Group > Group Org Export > Group Org Export General > Option Org Export With Sub Superscripts
After that, to show sub/super scripts, type these Sometext_{subscript} Sometext^notsuperscript^{superscript}
38.3.26. Include image
[[[[file:images/abc.jpg]]]] And C-c C-x C-v to toggle display images on Emacs
38.3.27. dot
- Install Graphviz
- Graphviz using .msi, copy
C:\Program Files (x86)\Graphviz2.38\binto PATH and make suredotordot.execan be run brew install graphviz
- Graphviz using .msi, copy
- https://vxlabs.com/2014/12/04/inline-graphviz-dot-evaluation-for-graphs-using-emacs-org-mode-and-org-babel/
And add this to .emacs
(org-babel-do-load-languages 'org-babel-load-languages '((dot . t)))
To compile C-c C-c and display inline image C-c C-x C-v
Graphviz Node, Edge and Graph Attributes
- rankdir
- default layout is from top to bottom. LR is left to right
- splines
- how edges look
noneor""- no edges
falseorline- edges can go through nodes
trueorspline- edges are routed around nodes (splines) and edges can be curved in order to avoid going through nodes
curved- edges are curved arcs
polylines- edges are one or multiple straight lines connected
ortho- edges are routed with 2 perpendicular straight lines
38.4. VS Code
38.4.1. Install, Basics
- Ubuntu
sudo snap install --classic code- (no term)
- Mac
brew install --cask visual-studio-code- C-S-p, select
Shell Command: Install 'code' command in PATH
- Updates and Blogs
- https://code.visualstudio.com/updates
- On Windows, I found I need to admin open to update VS Code
38.4.2. Extensions
ext install extension.name- You can disable one extension on User level and just enable it for the current workspace
38.4.2.2. Live Server
38.4.2.3. Live Share
38.4.2.4. Remote SSH ms-vscode-remote.remote-ssh
- Basics
- Basics
- Install SSH client on local machine. Run command in VS Code
Remote-SSH: Connect to Host... - After successful login, open Remote Explorer in left View Bar
- Set up forward port
- File > Close Remote Connection
- Install SSH client on local machine. Run command in VS Code
- Install Local Extensions on Remote
- Remote Explorer > LOCAL - INSTALLED, the cloud download icon
- Limitations
- Alpine Linux and non-glibc based Linux SSH remote host is not supported
- https://code.visualstudio.com/docs/remote/ssh
- https://code.visualstudio.com/docs/remote/troubleshooting#_ssh-tips
- Some files including extensions are installed in remote host
~/.vscode-server. Local VS Code is at~/.vscode - VS Code nees execute access on
/tmpdirectory on remote host
- Some files including extensions are installed in remote host
- Basics
- Local > Vagrant > Docker Container
- Goal
- VS Code remote development on a Docker container, which is spawn by Vagrant (Docker host) using Remote - SSH VS Code extension
- (no term)
- Install linux:ssh:install in the target container
- linux:ssh:permitrootlogin to yes
- linux:ssh:sshd_config:port to 22
- linux:ssh:sshd_config:passwordauthentication to yes
- Set a password for root linux:user:passwd
- linux:ssh:restart
- (no term)
- On Docker host (Vagrant)
- Get docker container ip docker:inspect:ip
- Test
ssh -p 22 root@1.2.3.4 - Set up linux:ssh:config file, and linux:ssh:public-private keys, so that docker host can run
ssh onedockercontainerto connect - Test
ssh onedockercontainer
- (no term)
- On Windows
- Copy the private key generated on Docker host to Windows
~/.ssh/id_rsa_vagrant_spawn - vagrant:ssh:pure ssh and test
ssh myvagrant ~/.ssh/config# This is added by `vagrant ssh-config --host myvagrant >> ~/.ssh/config` Host myvagrant HostName 127.0.0.1 User vagrant Port 2222 UserKnownHostsFile /dev/null StrictHostKeyChecking no PasswordAuthentication no IdentityFile C:/c/.vagrant/machines/default/virtualbox/private_key IdentitiesOnly yes LogLevel FATAL Host remotecontainer Hostname 172.17.0.2 User root # the private key is copied from Docker host (Vagrant) IdentityFile ~/.ssh/id_rsa_vagrant_spawn # On Windows after `ssh -V` is v8+, just use `ssh` not the whole path of the executable ProxyCommand C:\WINDOWS\System32\OpenSSH\ssh.exe myvagrant -W %h:%p Host remotecontainer2 Hostname 172.17.0.2 User root IdentityFile ~/.ssh/id_rsa_vagrant_spawn # On Windows after `ssh -V` is v8+, use ProxyJump ProxyJump myvagrant
- done!
- Copy the private key generated on Docker host to Windows
- (no term)
- TS: may need to linux:ssh:known_hosts:remove
38.4.2.5. Rest Client humao.rest-clident
- Create
.httpfile withGET http://a.ca HTTP/1.1 - Generate target code to execute that REST request
38.4.2.6. Better Comments aaron-bond.better-comments
38.4.2.7. Prettier - Code formatter
38.4.2.8. Bracket Pair Colorizer 2
38.4.2.9. Python extension for Visual Studio Code vs:code:ext:my-python.python
- Select code and run
S-Enter- (no term)
- Linting
- Default is PyLint
- Change to PEP8
pip install autopep8- In VS Code,
C-,searchpython formatting providerchange it to autopep8
- (no term)
- Command
- Select interpreter
- Run Python File in Terminal
38.4.2.10. Python Docstring Generator njpwerner.autodocstring
38.4.2.11. JavaScript (ES6) code snippets xabikos.javascriptsnippets
38.4.2.12. IntelliJ IDEA Keybindings
- name
- k–kato.intellij-idea-keybindings
38.4.2.13. PHP related extensions
- PHP Tools for Visual Studio Code
- https://marketplace.visualstudio.com/items?itemName=DEVSENSE.phptools-vscode
- https://docs.devsense.com/en/vscode/php-version
Executable
#!/bin/bash docker exec -i existing-docker-container-name php "$@"
- PHP Intelephense
- Follow the Quick Start to disable some builtin extensions
- PHPDoc Comment
- Shortcut
- Add PHPDoc Comment
⇧ ⌘ i
- Shortcut
38.4.2.14. ESLint
- See here for PhpStorm and ESLint
- https://www.jetbrains.com/help/phpstorm/eslint.html
# package.json is required npm init --yes # Install eslint and corresponding standards into ./node_modules and generate .eslintrc.json file npx eslint --init
38.4.2.15. npm Intellisense christian-kohler.npm-intellisense
Autocomplete npm modules in import statements based on package.json
38.4.2.16. Import Cost wix.vscode-import-cost
38.4.2.17. stylelint
38.4.2.18. Glitch for VS Code vs:code:ext:glitch.glitch
- https://marketplace.visualstudio.com/items?itemName=glitch.glitch
- glitch show commands
- Sign in
- Open Project
38.4.2.19. Sass syler.sass-indented
38.4.2.20. MJML vs:code:ext:attilabuti.vscode-mjml
38.4.3. View
38.4.3.1. Editor Layout
38.4.4. Shortcuts
- Command Pallete / Show All Commands
C-S-por⇧ ⌘ p- (no term)
- Left bar is called View Bar that contains views
- Explorer
C-S-eor⇧ ⌘ e- Debug
- C-S-d
- Version control
- C-S-g
- Search
- C-S-f
- Extension
- C-S-x
- Toggle Side Bar Visibility
C-b
- (no term)
- View
- Toggle Zen mode
C-k z
- (no term)
- Navigate Cursor
- Go to Bracket (jump to open or end bracket)
C-S-\- Go to
C-g- Go to Symbol in File
C-S-o
- (no term)
- Navigate Files, Folders, Editors
- Cycle through opened tabs
C-TabquickopennavigateNextInEditorPicker- Close Editor
C-w- Reopen Closed Editor
C-S-t- Toggle Vertical/Horizontal Editor Layout
M-S-0- Split Editor
C-\- Go to File (search file)
C-p- Go to Definition
F12orC-LeftClick- Go to Symbol in Workspace
C-t- Go Backward / Forward
M-LeftM-Right- Find All References (in opened files only..)
M-S-F12
- (no term)
- Edit
- Indent / Outdent
C-[C-]- Format Selection
C-k C-f- Expand Selection
M-S-RightArrow- Move Line Up (move selection up)
M-UpArrow- Delete line
C-S-k- Trigger Suggest
C-Space- Toggle Block Comment
M-S-a- Toggle Line Comment
C-/- (no term)
- Multiple cursors
- Add Selection to Next Find Match
C-d, hit again will create another cursor which selects the same word- Select All Occurances of Find Match
C-S-l, select and create cursors for all matches in file- Create a cursor at caret
M-LeftClick- Add Cursor Above / below
C-M-Up- (no term)
- Left or right to move the cursors
- Right click on a variable name, Rename Symbol
F2
- (no term)
- Search
Search: Find in FilesC-S-f- While in view, see which files are included/excluded
C-S-jsearch.toggleQueryDetails
- (no term)
- Terminal
- Toggle Integrated Terminal
C-`- Scroll Up (Line)
C-M-PageUp
- Focus on Editor Group 1
C-1View: Focus First Editor Group
38.4.5. Setting
- Settings
C-,- User settings
- saved in
- Windows
~/.config/Code/User/settings.jsonor~/AppData/Roaming/Code/User/settings.json- (no term)
- Mac
- User
~/Library/Application Support/Code/User/settings.json- Workspace
.vscode/settings.jsonat project folder- Open settings.json from VSCode
⌘ ,select User or Workspace, and in the right top corner there's an icon to open
- (no term)
- Open the settings.json,
C-S-psearchPreferences: Configure Language Specific Settings
- (no term)
- Emmet
- emmet.triggerExpansionOnTab
true- includeLanguages
{ "vue-html": "html", "mjml": "mjml" }
- (no term)
- Editor
- showFoldingControls
- always
- wordWrap
on- formatOnType
- true
- formatOnPaste
- true
38.5. WinMerge
- Edit > Options
- Backup Files
- Create backup files into Global backup folder
- Backup Files
38.6. WinSCP
38.6.1. Connect to remote using SSH
- New Session > New Site
- SFTP
- IP or full domain name, no alias
- empty
- Advanced > SSH Authentication > Private key file
- it will generate if the private key is not Putty style e.g. id_rsa to id_rsa.ppk
- Save the new site
- Login!
38.6.2. Connect to remote using SSH with sudo su required
- New Session > New Site
- SCP
- Host name, Port Number, Username
- IP or full domain name, no alias
- Password
- empty
- (no term)
Advanced
- SSH Authentication > Private key file
- it will generate if the private key is not Putty style e.g. id_rsa to id_rsa.ppk
- Environment > Directories > Remote directory
- change to
/root - SCP/Shell
sudo su -
- (no term)
- Save the new site
- (no term)
- Login!
- SFTP
- Host name, Port Number, Username
- IP or full domain name, no alias
- Password
- empty
- (no term)
Advanced
- SSH Authentication > Private key file
- it will generate if the private key is not Putty style e.g. id_rsa to id_rsa.ppk
- Environment > Directories > Remote directory
- change to
/root - SFTP > Protocol options > SFTP server
sudo su -c /usr/lib/sftp-server. Tested on AWS Ubuntu 16 and claimed working for 18
- (no term)
- Save the new site
- (no term)
- Login!
38.6.3. VS Code Extension alternative!
- Extension name
- SSH FS
- (no term)
- An icon will be created on the left nav bar
- Setup an SSH connection. OpenSSH config cannot be pulled, so manual config is required
- Extensions > this extension > setting icon
The settings.json is shared by VS Code as a whole, here's a sample of how 2 SSH FS hosts set up look like
{ "emmet.triggerExpansionOnTab": true, "editor.showFoldingControls": "always", "editor.wordWrap": "on", "glitch.token": "1700535f-5133-41b1-81a2-7520deb26f75", "editor.fontFamily": "Source Code Pro", "editor.fontLigatures": true, "editor.fontSize": 13, "editor.formatOnType": true, "editor.formatOnPaste": true, "editor.renderControlCharacters": false, "editor.renderWhitespace": "all", "sshfs.configs": [ { "name": "app1", "host": "1.2.3.5", "port": 22, "username": "nonrootuser", "privateKeyPath": "$HOME/.ssh/id_rsa", "sftpSudo": true }, { "name": "app2", "host": "1.2.3.4", "username": "nonrootuser", "privateKeyPath": "$HOME/.ssh/id_rsa_2", "sftpSudo": true } ], }
38.7. Regex
- Regular expression is also called pattern. Syntax
- Boolean "or"
|- Grouping
()- Quantification
- quantifiers
?*+- e.g.
{2}2 times {2,}minimum 2 times{2,4}between 2 or 4 times
- (no term)
- Wildcard
- Online Tester
38.8. Chrome
38.8.1. Resources
- Blog (new releases)
- https://developer.chrome.com/blog/
38.8.2. Command Menu (DevTools > C-S P)
- Coverage (Show coverage)
- JavaScript, CSS file unused bytes
- Screenshot
- full size screenshot
38.8.3. Toggle/detach dock side (DevTools > C-S D)
38.8.4. Navigate Panels (DevTools > C [ or C ])
38.8.5. Navigate among tabs ⌘ ⇧ a ⌥ ⌘ left/right
38.8.6. Console Panel
38.8.6.1. Console Commands
https://developers.google.com/web/tools/chrome-devtools/console/console-reference
console.count
function login(user) { console.count("Login called for user " + user); } users = [ // by last name since we have too many Pauls. 'Irish', 'Bakaus', 'Kinlan' ]; users.forEach(function(element, index, array) { login(element); }); login(users[0]);
- Formats DOM elements as JavaScript objects
- can be an array, object or an array of objects
console.assert( condition, string )Show string only when the first parameter evaluates to falseconsole.assert(list.childNodes.length <= 500, "Node count is > 500");Specifiers
//Style console output with CSS console.log("%cThis will be formatted with large, blue text", "color: blue; font-size: x-large"); // format a value as string console.log("%s has %d points", "Sam", 100); console.group("Authenticating user '%s'", user); console.group("Authorizing user '%s'", user); console.log("User '%s' was authorized.", user); console.groupEnd(); console.groupEnd();
- %i or %d
- integer
- %f
- float
- %o
- expandable DOM element as seen in Elements panel
- %O
- Formats the value as an expandable JavaScript object
- %c
- color and other styles
console.warn(),console.info(),console.error()console.trace(object)
38.8.6.2. Command Line API
$('code') // Returns the first code element in the document $$('figure') // Returns an array of all figure elements in the document $x('html/body/p') // Returns an array of all paragraphs in the document body. XPath $('[data-target="inspecting-dom-elements-example"]') // CSS selector // $_ :: last evaluated expression // Open the selected element on Elements Panel inspect($_) // $0, $1, $2, $3, $4 :: recently selected // copy a JavaScript variable value, later paste to e.g. Notepad copy(abc) monitorEvents(document.body, "click");
38.8.6.3. Inject CSS file in Console
jQuery(document.head).append('<link rel="stylesheet" href="path_to_my_css">');
38.8.6.4. Offload CSS file in Console
document.styleSheets[0].disabled = true; $0.disabled = true; // select the <link> or <style> element in Elements and then run in Console
38.8.7. Block Request or Domain - Network
Block a certain file request or any files from a domain. Network > right click on a file and Block Request
38.8.8. Get all events of an element in Console
getEventListeners($0);
38.8.9. XPath XPath
- In DevTools > Elements, search using Xpath, e.g. find all h4 element
//h4 - xPath Online tools
- select all nodes with name
- select from the root node
- select nodes from the current node that match the select no matter where they are
//book- select all book elements in the whole XML document
.//book- select all book elements which are under the current node
- select the current node
- select the parent of the current node
- select attribute
- first book element
- last book element
- second last book element
- first 2 book elements
- all book elements that have price element with a value greater than 35.00
- select lang attributes of all title elements in book elements. Not filter!
- match any element node
- match any attribute node. Have at least one attribute
- select multiple paths
38.8.9.1. Predicates
- Things inside
[]are called predicates - Multiple predicates
//*[local-name()='CSItem'][@name='NameAsReported'][@maxLength='250'], or//*[local-name()='CSItem' and @name='NameAsReported' and @maxLength='250']//*[local-name()='CSItem' and @name='NameAsReported' and @maxLength='250'][1]- Select the first CSItem element
//*[local-name()='CSItem' and @name='NameAsReported' and @maxLength='250'][1]/@description- Select the description attribute of the first CSItem element
//library/books/book[author[firstName[contains(text(),'Li')]]]//library/books/*[local-name()='book'][*[local-name()='author'][*[local-name()='firstName' and contains(text(), 'Li')]]]- using local-name
- (no term)
- Select all book elements that that have author elements that have firstName elements that have text Li
//library/books/book/author/firstName[contains(text(),'Li')]or//library/books/*[local-name()='book']/*[local-name()='author']/*[local-name()='firstName' and contains(text(), 'Li')]- Select all firstName elements (which has text Li) of the author elememts of the book elements
- Functions
- contains()
//author[firstName[contains(text(),'Li')]]- text()
- text of a node
- (no term)
- starts-with()
//author[starts-with(@firstName, 'Li')]- attribute value starts with
38.8.9.2. Axis Syntax
axisname::nodetest[predicate]- axisname is the relationship name
- self
- child
- nodetest is a node name to test
child::book- book nodes that are children of the current node
attribute::lang- lang attribute of the current node
child::*- children elements of the current node
child::text()- text node of the current node
- (no term)
child::node()
38.8.9.3. XPath Operators
+,-,*,div- division remainder e.g.
5 mod 2 - combine 2 node-sets
=,!=,<,<=,>,>=price=9.8 or price=9.7and
38.8.10. Turn on DevTools Experiment
chrome://flags > search a Feature, enable and relaunch
Developer Tools experiments :: Experiments under DevTools > Settings Experimental JavaScript
38.8.11. Elements Panel
38.8.11.1. Styles
Click on color and expand Contrast Ratio produces 1 line. Choose color below the line in order to make it AA or AAA compliant.
38.8.11.2. Accessibility
38.8.12. Audits Panel
Run checks using Light House
38.8.13. Sources Panel
38.8.13.1. Local Overrides
Specify a directory where DevTools should save changes. This directory will contain all changes across different website domains. DevTools > Sources > Overrides
Select folder > Allow > Enable Local Overrides
This only works to override styles that are loaded by a CSS file and you can override the CSS files.
38.8.13.2. Stop running script
- For example, a JavaScript event (onmouseover) creates and displays a new div. We want to stop and investigate on the new div
- This is different from adding
:hoveras it's for CSS only - Open DevTools, got to Sources panel, hover on the element, hit
F8, move mouse to Sources panel, you will find the script is stopped running- Hit
F8again to resume script running
- Hit
38.8.13.3. Event Listener Breakpoints and Blackboxing
Mouse > click Say it jumps jquery.js at function abc at line 1000. Find this under Call Stack, right click and Blackbox script. jquery.js will be ignored.
Right click again to stop blackboxing, To see all blackboxed scripts: DevTools > Settings > Blackboxing https://developer.chrome.com/devtools/docs/blackboxing
38.8.14. Extension
38.8.14.1. Google Tag Assistant chrome:google tag assistant
38.8.14.2. Postman
- Raw request and response
- Postman console (icon at the bottom)
- POST key value is array
- create multiple keys e.g.
- key
channel[0], valuehello - key
channel[1], valueworld
- key
- Variable Scopes
- https://learning.postman.com/docs/sending-requests/variables/
- a workspace which has collections, requests, test scripts and environments
- throughout the requests in a collection and are independent of environments
- e.g. local vs testing or production. Only one environment can be active at a time
- only in request scripts
- come from external CSV and JSON files to define data sets
pm.globals.set("global_variable_key", "variable_value"); pm.collectionVariables.set("collection_variable_key", "variable_value"); pm.environment.set("environment_variable_key", "variable_value"); pm.variables.set("local_variable_key", "variable_value"); pm.environment.unset("variable_key"); // {{local_variable_key}} in GUI pm.variables.get("variable_key"); //access a variable at any scope including local pm.globals.get("variable_key"); //access a global variable pm.collectionVariables.get("variable_key"); //access a collection variable pm.environment.get("variable_key"); //access an environment variable
- Tests tab for each request
- It's run after the request has been sent and has response
- https://postman-quick-reference-guide.readthedocs.io/en/latest/assertions.html
- Pre-request Script for each request
- Moment.js
const moment = require('moment'); // moment() === '2021-07-28T19:26:01.558Z' // UTC pm.variables.set("mydate", moment().format("YYYY-MM-DD-HH")); // eq. to PHP Y-m-d-H. Time is converted to current timezone! // So if local timezone is America/Toronto then it's 2021-07-28-15 pm.sendRequest("http://worldtimeapi.org/api/timezone/America/Toronto", function (err, res) { pm.globals.set("localTime", res.json().datetime); // 2021-07-28T15:15:18.644592-04:00 }); var localTime = moment(pm.variables.get('localTime')).format("YYYY-MM-DD-HH"); // 2021-07-28-15 console.log(moment.utc().format("YYYY-MM-DD-HH")); // 2021-07-28-19
- CryptoJS
// no need to require // eq. to PHP sha1(key) var key = 'abc'; var hash = CryptoJS.SHA1(key); console.info(hash.toString(CryptoJS.enc.Hex));
- Moment.js
- Collection run
- Setup Tests, Pre-script for each request
- Select the requests and run them in sequence
- Pricing and Plans
- Plans are for a Team
- Free
- Any sort of collaboration e.g. Collections for up to 3 users
- Basics
- Professional
- Enterprise
- Free
- Plans are for a Team
- Team
- An individual Postman account can create a Team and then the individual can pay for a subscription plan or request someone else to pay
- Once the individual account (li@me.com) creates a Team (Li Team), the individual account becomes a Team account and all personal workspaces are under the Team account
- To check if you have an individual account, click on the avatar on the top right, in between
View ProfileandTeams(aboveSettings), you will see li@me.com with subtitleIndividual
- To check if you have an individual account, click on the avatar on the top right, in between
- li@me.com can join other Teams, but li@me.com still does not have an individual account
- li@me.com now joins with Li Team (as a creator) and Li2 Team
- When li@me.com leaves a Team (say Li2 Team) and li@me.com does not have an individual account, an option is provided to keep the personal workspaces of that Team in an individual account. Now li@me.com has an individual account
- Now li@me.com leaves Li Team, because li@me.com now has an individual account, no option is provided to keep the personal workspaces of Li Team to the individual account. You will have to export data prior to leaving.
- Once the individual account (li@me.com) creates a Team (Li Team), the individual account becomes a Team account and all personal workspaces are under the Team account
- An individual Postman account can create a Workspace that has Visibility = Team
- Visibility = Partner requires subscription plan Enterprise. It enables
- And then the Workspace is available for members of the Team
- An individual Postman account can create a Team and then the individual can pay for a subscription plan or request someone else to pay
38.8.14.3. Save Page WE
Chrome's default save page as HTML has some limitations such as not all images are downloaded. This saves all content in one HTML file.
Font file is not converted but you can fix the url() manually.
Turn off 38.8.24
38.8.14.4. Ignore X-Frame headers
Ignore headers X-Frame-x e.g. X-Frame-Options
38.8.14.5. Requestly
Modify http request and response headers
38.8.14.6. jQuery Audit
Find function that attaches events to a selected element. Element > jQuery Audit Expand the Events and RC on handler to Reveal function in source.
38.8.14.7. Web Server for Chrome
38.8.14.8. Wasp chrome:wasp
- Web Analytics Solution Profiler
- DevTools > beside Audits panel, there's a WASP panel
- WASP tracks across page opening history (continously)
- Scripts
- Loaded JavaScript files
- Show the path of JS files and may be cookies and headers that are set by the script
- Tags
- Tags are scripts that are recognized by WASP and it has deeper data
- Tags that are triggered in GTM also appear here
- googletagmanager.com
- before googletagservices.com
- google:dfp, google:gam, gpt.js
- google-analytics.com (analytics.js) ga:ga
- googleadservices.com (old AdWords)
38.8.14.9. Hosts file
38.8.14.12. React Developer Tools
- DevTools > tabs at top Components and Profiler
- Extension Setting
- Allow access to file URLs
- Profiler tab
- Recording needs to be turned on. Each event will be recorded separately e.g. a button is clicked twice and hence 2 profiling recordings
- Setting
- General
- Highlight updates when components render
- If a component is partially updated, the whole component is flashed. Enable Chrome > More Tools > Rendering > Paint flashing to see which DOM elements in the component are re-rendered
- Highlight updates when components render
- Profiler
- Record why each component rendered while profiling
- General
38.8.15. Network
38.8.15.1. Filter Post Requests
method:POST
38.8.15.2. Preserve log
- Preserve all requests log even though the page URL is changed (e.g. after the Post request, URL changes)
- Change of this setting will carry to all new DevTools sessions
- For links that open in another tab, tell DevTools to auto open and hence all Network is tracked in the new tab/window
- DevTools > F1 (Settings > Preferences > Global) > Auto-open DevTools for popups
- If the parent tab does not have DevTools open, the new tab will not open the DevTools
- You may need to enable Preserve Log in the parent tab first. But after that, whether or not Preserve Log on the parent tab does not matter.
38.8.15.3. Capture Screeshots
Double click to show image and single click to review timeline
38.8.15.4. Network Performance
Blue line is DOMContentLoaded and red line is load
In Waterfall for a request
- Queueing
- The browser queues requests when:
- There are higher priority requests.
- There are already six TCP connections open for this origin, which is the limit. Applies to HTTP/1.0 and HTTP/1.1 only.
- The browser is briefly allocating space in the disk cache
- Stalled
- The request could be stalled for any of the reasons described in Queueing.
- DNS Lookup
- The browser is resolving the request's IP address.
- Proxy negotiation
- The browser is negotiating the request with a proxy server.
- (no term)
- Request sent. The request is being sent.
- (no term)
- ServiceWorker Preparation. The browser is starting up the service worker.
- (no term)
- Request to ServiceWorker. The request is being sent to the service worker.
- (no term)
- Waiting (TTFB). The browser is waiting for the first byte of a response. TTFB stands for Time To First Byte. This timing includes 1 round trip of latency and the time the server took to prepare the response.
- (no term)
- Content Download. The browser is receiving the response.
- (no term)
- Receiving Push. The browser is receiving data for this response via HTTP/2 Server Push.
- (no term)
- Reading Push. The browser is reading the local data previously received.
38.8.16. Change User Agent
F12 or C-S-i > 3 dots > More Tools > Network conditions > User agent > Chrome - Windows
38.8.17. Which DOM elements are repainted
F12 > 3 dots > More Tools > Rendering > Paint flashing
38.8.18. Performance
38.8.18.1. Memory
- https://developer.chrome.com/docs/devtools/memory-problems/
- Real time memory usage for a tab using Task Manager
- 3 dots in the main window (not DevTools) > More Tools > Task Manager, right click on columns and show JavaScript Memory
- Real time memory usage, CPU, number of DOM nodes and more using Performance monitor drawer
- Drawer is in the Console Drawer
- See Heap size by recording profiling in Time Line format in Performance Panel
- DevTools > Command Menu > type
Show Performanceand go to Performance Panel > enable Memory and hitStart Profiling and reload page- Don't forget to hit the recycle button to force collect garbage when starting and ending a recording
- DevTools > Command Menu > type
- See Heap Snapshots in Memory Panel
- DevTools > Command Menu > type
Show Memoryand go toMemoryPanel
- DevTools > Command Menu > type
38.8.18.2. Performance Insights Panel
⌘ ⇧ popen Command Menu and typeperformanceto open thePerformance Insightspanel- https://developer.chrome.com/docs/devtools/performance-insights/
- Use W to zoom in, S to zoom out, A to go to left of the timeline, D to go to right
- Can record or measure page load
- Major events like Rendering blocking requests, long tasks and layout shifts are laid out in a vertical timeline. Those are usually the ones need special attention (insights) and details like call tree are available. Not all events/calls have details. For full details, record in the Performance Panel.
- Key metrics in Timeline
- First Contentful Paint (FCP)
- DOM Content Loaded (DCL)
- Largest Contentful Paint (LCP)
- Cumulative Layout Shift (CLS)
38.8.18.3. Performance Panel
- Command Menu > type
performanceto openPerformancepanel - https://developer.chrome.com/docs/devtools/performance/reference/
- Don't forget to force collect garbage! See Chrome Monitor Memory
- The first chart on top is called Overview and sections follow
- Select a section and use W S A D to move the zoom in, move left, zoom out and move right in Overview.
- Selected section or item will be outlined in blue.
- Only the
Mainsection shows JS Heap, Documents, Nodes, Listeners, GPU Memory
^E.*in regex e.g. Evaluate Script…- Shift+Enter to select the previous and Enter for the next.
- Sections
- Network
- HTML: Blue
- CSS: Purple
- JS: Yellow
- Images: Green
- Network
- Event, Function Call, Evaluate Script, Compile Script, etc.
- Recalculate Style, Layout, Update Layer Tree, etc.
- Under each section, the chart is called flame chart. Root activities are the ones at the top of Main section chart and they are activities that cause the browser to do some work.
- Select a section and use W S A D to move the zoom in, move left, zoom out and move right in Overview.
- Tabs
- Call Tree
- Which root activities cause the most work during the selected portion of a recording (range in Overview)
- Columns
- Activity
- root activities followed by their children
- Self Time
- the time directly spent in that activity.
- Total Time
- the time spent in that activity or any of its children
- On the top right, there's a button Show Heaviest Stack
- shows which children of the selected activity took the longest time to execute
- Bottom-Up
- Which activities directly took up the most time in aggregate
- Event Log
- List activites in the order in which they occurred during the recording
- Call Tree
38.8.18.4. Recorder Panel (experimental)
- Command Menu > type
Show Recorderand go to Recorder Panel - Start recording and perform user actions e.g. go to a page, click a button. Then stop recording
- For each step,
Add durationto make the step lasts longer (delay the time to go to next step) - Clicking
Performance panelwill perform the recorded and check out the Performance profiling recording!
38.8.19. Clear cache for one domain
Settings > Advanced settings > Content Settings under Privacy > All cookies and site data under Cookies > search a website and clear
38.8.20. Refresh Favicon
Delete files Favicons and Favicons-journal in this folder. It will remove all favicon cache
C:\Users\your_username\AppData\Local\Google\Chrome\User Data\Default
38.8.21. Flush DNS Caches in Chrome
Even ipconfig /flushdns is run, Chrome still caches the DNS
Navigate to chrome://net-internals/#dns and press "Clear host cache" button
Or on the top right, down arrow > Tools > Clear Cache
Make sure bookmark has the right url. Say you bookmark abc.com as www.abc.com, if you type abc.com, Chrome will go to www.abc.com!
Also say before there was a DNS A record for www.abc.com to 1.2.3.4 which is an old IP. You remove it and set a CNAME for www.abc.com to abc.com, www.abc.com and https://www.abc.com will actually go to 1.2.3.4 which is not expected.
For more edge cases like above and in case of the local network DNS caches which you have no control of, set the network adapter to use 8.8.8.8 Google DNS.
38.8.22. Flash Couldn't Load Plugin
Properites > Security, give Full Control for Everyone in this folder C:\Users\you\AppData\Local\Google\Chrome\User Data\PepperFlash\some.version.number
38.8.23. Allow js:xmlhttprequest for local files
38.8.24. Allow CORS locally
- May need to close all Chrome instances. Mine opens the Chromium version not the regular Chrome.
chrome.exe --disable-web-security --user-data-diropen -n -a /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --args --user-data-dir="/tmp/chrome_dev_test" --disable-web-security- Errors such as
- Access to Font at 'yourwebsite.com' from origin 'e.g. localhost' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'e.g. localhost' is therefore not allowed access.
- A CORS request is an HTTP request that includes an
Originheader. When do browsers send out CORS requests?- Requests that are neither
GETnorHEAD- All POST requests
- for same domain requests, browsers may not add
Originheader in requests- Non
OPTIONSrequests
- Non
- Browsers require the responses for CORS requests to have
Access-Control-Allow-Originheader - Some
GETrequests that have Origin header set - About OPTIONS or preflight request
- Requests that are neither
38.8.25. Remote Debug Android Devices
On Android, Settings > System > About phone > tap Build number 7 times, then Sytem > Developer options is enabled, enable USB debugging. It allows Android Studio and other SDK tools to recognize your device when connected via USB, so you can use the debugger and other tools. Other options on Android: https://developer.android.com/studio/debug/dev-options
Connect Android to test machine > DevTools > 3 dots > More tools > Remote devices > enable Discover USB devices.
When the phone is connected to a computer, on the phone say transfer photos and open the device on the computer so that the debug connection can be made.
On the phone, you may need to enable Always prompt when connecting to USB in Developer Options in order for prompting Use USB for Transfer Photos.
Don't let the phone screen turn off while debugging.
38.8.26. Debug
- Add
debugger;to JavaScript so that Chrome stops there as a breakpoint Within an HTML
<script>tag, set the name of the file so that it can be opened under Sources tab<script> ... (your code here) //# sourceURL=somename.js </script>
38.8.27. Command line
https://peter.sh/experiments/chromium-command-line-switches/
chrome.exe [option]... chromium-browser [option]...
For headless
- –headless
- need to have it when CLI is running by a root user
- By default,
file://URIs cannot read otherfile://URIs e.g. js:xmlhttprequest for local files - –disable-web-security
- Don't enforce the same-origin policy
- See 38.8.24
38.8.28. Security chrome://settings/security
- Enhanced protection vs Standard protection
38.9. Firefox
38.9.1. Make all new windows open in Private Mode
- Visit about:config and click "Yes, I'll be careful"
- Search for
browser.privatebrowsing.autostart - Change value to True
38.10. BrowserStack
2018/03 Devices chosen for screenshots
iOS v8.3 iPad Mini 2, iPhone 6, iPhone 6 Plus v7 iPhone 5S, iPad Mini v6 iPad 3
Android Samsung S5, S4 Google Nexus 9, 7
Win 10 Edge 15, IE 11, Chrome 50
Win 8.1 IE 11
Win 8 IE 10
Win 7 IE 11, 10
Mac OS X 10.11 El Capitan Safari 9.1 Chrome 50
Mac OS X 10.10 Yosemite Safari 8 Chrome 50
Mac OS X 10.9 Mavericks Safari 7.1
38.11. Device Market share, Metrics
38.11.1. Browser
https://docs.google.com/spreadsheets/d/1iAOjjA-MG5nUiAidhP_MdmnM-kEbX1j6qpMI9zbKjCw/edit?usp=sharing
https://data.apteligent.com/ios/ iOS Overview 2018/03 <=6 .06% (.06%) <=7 .30% (.23%) <=8 .93% (.63%) <=9 6.5% (5.57%) <=10 20.5% (13.94%)
https://data.apteligent.com/android/ Android overview 2018/03 <=4 .41% 4.4 12%
https://netmarketshare.com Browser Market Share across all devices 2018/03 Chrome 58.57% Safari 17.15% Firefox 6.61% Internet Explorer 6.52% Edge 2.11%
Internet Explorer Browser Market Share across all devices 2018/03 IE 11 71.76% IE 10 5.5% IE 9 6.81% IE 8 12.19% IE 7 1.56%
http://gs.statcounter.com/os-version-market-share/windows/desktop/worldwide Windows Desktop 2018/03 Win 10 38.29% Win 7 44.58% Win 8.1 9.21% Win 8 2.5% Win XP 4.47% Win Vista .86%
http://gs.statcounter.com/macos-version-market-share/desktop/worldwide macOS Desktop overview 2018/03 High Sierra 10.13 14.75% Sierra 10.12 39.27% El Capitan 10.11 21.22% Yosemite 10.10 14.12% Mavericks 10.9 4.95% Mountain Lion 10.8 < 2.13% Lion 10.7 <2.13% Snow Leopard 10.6 2.13%
Resolution Distribution http://gs.statcounter.com/
38.11.2. Device Metrics tool:device metrics
- https://material.io/tools/devices/
- Device Platform, Screen dimensions in in or cm, aspect ratio, dp or dip (density-indepenndent pixels), px (CSS pixels), density (px/dp)
- https://www.mydevice.io/
- newer devices
- http://screensiz.es/
- By popularity
- http://viewportsizes.com/
- Old source. By release date
- Device and viewport size in JavaScript
- http://ryanve.com/lab/dimensions/
- (no term)
- iPhone X, iPhone 8 Plus, iPhone 7 Plus and iPhone 6s Plus has pixel density (scale factor) of 3x and others are 2x
- Android devices
- https://developer.android.com/about/dashboards/
38.11.3. Email Client Market Share
38.12. Windows
38.12.1. Win Shortcuts
- https://support.microsoft.com/en-ie/help/12445/windows-keyboard-shortcuts
- switch input language and keyboard layout
- minimize all windows except the active one
- stretch active window to top while maintaining width
- move active window to next monitor
- App switching
- Win + number
- open the app # pinned to the taskbar. If the app is already running, switch to that app
- Win + Shift + number
- same as above but open a new instance
- Win + Ctrl + Shift + number
- same as above but open a new instance as Administrator
- Win + Ctrl + number
- switch to the last active window of the app pinned to the taskbar
- Win + Alt + number
- same as above but open the Jump List
- Win + T
- cycle through apps on taskbar
Alt + Esc- cycle through apps in the order in which they were opened
- temp peak at desktop
- display System Properties
- open Windows Settings or go back to the current Windows Settings
- enlarge text on display, enable night light mode
Win + P: presentation (change monitors)- Connect to other devices on network
- Open Cortana in listening mode. Off by default. Start > Settings > Cortana and turn on "Let Cortana listen for my commands when I press the Windows logo key + C"
- snipping, screenshot
- Windows Whiteboard
- Still need to use snip and then copy and paste to Whiteboard
- File Explorer
Ctrl + RvsF5- refresh
Alt + Left/Right- Go back/forward
Alt + Up- Go up one folder level in File Explorer
F3- search in File Explorer
Alt + Enter- display properties for the selected item (right click menu)
Shift + F10- display shortcut menu for the selected item
Ctrl + DvsShift + delete- delete and move it to recycle bin vs permanent delete
- Inside a window or app
- F6
- cycle through screen elements in a window or desktop
- F10
- activate Menu in active app
Alt + Spacebar- open shortcut menu like Close, Maximize, Minimize the active app
Ctrl + F4vsAlt + F4- close current document while the latter closes the whole app
Ctrl + YvsCtrl + Z- Redo and undo
- Typing
Win + .- display Emoji
Ctrl + Left/Right- move cursor to the start or end of the next word
Ctrl + Up/Down- move cursor to the prev/next paragraph
- resize Start menu when it's open
- Remote Desktop Connection
- Swtich between host and remote
C-M-Breakthen useM-Tabas usual to switch between apps on host
38.12.2. File Explorer
38.12.2.1. Search
(kind:pics OR kind:videos) AND size:>100KB (*.jpg OR *.jpeg) AND size:>100KB AND (NOT tags:"to backup")
38.12.3. Win 7
Install KB3138612 and KB3145739 to boost Windows Update
38.12.4. Win 10
38.12.4.1. Boot to Safe Mode with Boot Options Menu
Start Menu > Power > hold Shift and click Restart
Or use command line where you need to wait 1 minute shutdown.exe /r /o
38.12.4.2. Install a printer loop in "Type a printer name"
Run bcdedit.exe /set nointegritychecks on and restart Windows 10 to disable the driver signature enforcement
Or Shift click on Power > Restart, choose Troubleshoot > Startup Settings > Restart, after restart choose F7 to disable driver signature enforcement
38.12.4.3. Windows version
- CMD run
winverreturns Version number and OS Build Number verreturns OS Build Number- Version number
- 1607
- known as the Anniversary Update
- 1703
- Windows 10 Creators Update
38.12.4.4. Upgrade from Windows 7
Make sure to do all critical and optional Windows Updates to ensure all drivers are up-to-date. Use Lenovo System Update to update all drivers and BIOS. If you experience black or blank screen during reboot to install Win 10, go to BIOS and change Display setting to Integrated (disable Discrete). Reboot and do the upgrade again.
Create Installation Media USB drive to install Win 10 on a different PC or reinstal
38.12.4.5. Ubuntu Bash on Windows
It's also called Windows Bash Shell
Turn on Developer Mode (it's not necessary any more)
You need to have Win 10 Anniverary Update, Creators Update is recommended.
Make sure OS Build is not below 14393
Win + X > Settings > System > About
Win + X > Settings > Update and Security > For developers > Developer Mode
Enable Windows Subsystem for Linux feature Search Windows Features > turn on Windows Subsystem for Linux (beta)
Win + S and type microsoft store, search linux and you will see different distros. Choose and install Ubuntu or a specific Ubuntu version. Then Win + S or go to Start > Dashboard to find the one just installed. Follow the username and password setup:
Reboot then open a command prompt, run
bash and type y to install Ubuntu on Windows.If you see "Unsupported console settings", right click on the cmd top bar > Properties > uncheck Use legacy console
Create a UNIX user and password which have no relationship to Windows username and password The user will be added to sudo group and will signed-in automatically every Bash instance.
- Multiple distro bash can be installed
- List all installed Linux environments
wslconfig /l - Set default Linux environment
wslconfig /setdefault <name> - Run one of these to directly open a specific Linux environment
ubuntuopensuse-42sles-12 - Run a command on a specific Linux environment in PowerShell or Command Prompt
ubuntu -c apt-get moo - Run a command on the default Linux environment
bash -c "uname -a"
- List all installed Linux environments
- Run
wsl(orbashbut deprecated) to get to Bash on Windows - Upgrade Linux Environment
- Win + S, type bash or ubuntu, right click to uninstall. Choose Ubuntu in Microsoft Store to install the latest version
- Uninstall and install through commands
- Access Windows restricted folders
A Bash session with Windows admin privileges (run cmd as admin) may cd /mnt/c/Users/Administrator
which is a directory restricted by Windows.
While a Bash session without admin privileges would see Permission Denied.
- Open File Explorer while you are in Bash for Windows
cmd.exe /c start .- (no term)
- Run Windows command (e.g. to open IE)
/mnt/c/Prgram\ Files\ \(x86\)/Internet\ Explorer/iexplore.exe - (no term)
- Ubuntu is installed at
%localappdata%\lxss\, don't modify files using Windows tools and apps here! - (no term)
- Windows Drives, Files and File Permissions
Mount mnt/c or /mnt/e is drive C: and E:. Store files in any mount and modify files using Windows or Linux apps. It's not possible to modify file permissions under /mnt/(your_drive). scp from here to remote will have files/directories full permission! Change those on remote host after scp..
- Remote files can be copied to any places other than /mnt/(your_drive). But don't open these files using Windows apps!
Permissions can be playe around in any places other than /mnt/(your_drive)
In order to connect to shared network drive, run this to create a d drive on Ubuntu and map it to Windows d drive
sudo mkdir /mnt/d sudo mount -t drvfs D: /mnt/d
38.12.4.6. WSL 2
# Run as administrator Command Prompt ver # If Windows version is lower than 18917, then the old WSL 1 is being used. # Update the WSL kernel wsl --update # list all images and WSL version wsl --list --verbose # eq. wsl -l -v # Set all images to run on WSL 2 wsl --set-default-version 2 # Inside WSL, find the host (Windows) IP wsl.exe hostname -I
38.12.4.7. Win 10 Build Update History
38.12.4.8. Startup Programs and Change HOMEDRIVE, HOMEPATH
Run shell:startup to find out the folder and copy shortucts of programs
That will return the startup folder for your user account only.
To find out the startup folder for all users, run shell:common startup
Active Directory might override some environment variables such as HOMEDRIVE HOMEPATH
Create a batch file C:\Windows\System32\userinit.cmd
@ECHO OFF SET HOMEDRIVE=C: SET HOMEPATH=\Users\%USERNAME% SET HOMESHARE=\\localhost\C$\Users\%USERNAME% @START C:\Windows\system32\userinit.exe
Set the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Userinit to 'C:\Windows\System32\userinit.cmd,'
Userinit default value is 'C:\Windows\system32\userinit.exe,'
38.12.4.9. Remove a keyboard
Win+S > Region & language settings > Select language and options > remove the keyboard if you see it, if not add the same language/keyboard and remove it.
Set default keyboard :: Control Panel > Language > Advanced settings > Override for default input method
38.12.4.10. Move a window
Shift + right click on a tab in Task Bar and you should see Move.
38.12.5. Win Server 2003
38.12.5.1. System Information
run winmsd.exe and look for processor to find out the bit count
38.12.5.2. Update_Windows.exe error
It consumes a lot of CPU and slows down the server. 2017/07/07. It's a hack.. Solution: Go to C:\Windows\Debug, Tools > Folder Options > View > Uncheck Hide Protected Operating System Files The batch files are referring to bitcoin mining website.. Ctrl+Alt+Del > Task Manager to stop Update_Windows.exe, then delete Update_Windows.exe, debug.exe, debug.bat, dll.bat
38.12.6. PowerSehll
38.12.6.1. Active directory
# see if a user `matt` is locked out Get-ADUser matt -Properties * | Select-Object LockedOut
38.12.6.2. Get-Command
- See file path of a command like linux which command
Get-Command sshwindows:c:which
38.12.6.3. Get-Command - gcm
Get manual of a command e.g. Select-String
gcm Select-String | select -expand definition
38.12.6.4. Search text in files
Get-ChildItem -Path "C:\path" -recurse | Select-String -Pattern "find me" | group path | select -Unique name # dir is an alias of Get-ChildItem, sls is an alias of Select-String # Since -Path and -Pattern are the default parameters in Get-ChildItem and Select-String, they can be omitted Get-ChildItem "c:\path" -recurse | Select-String "find me" | group path | select -unique name # current directory is ".\" # Exclude some binary files by file extension so that it runs faster dir ".\" -recurse -Exclude *jpg, *jpeg, *png, *pdf | sls "<body>" | group path | select -unique name # also exclude a folder, ? is alias of where dir ".\" -recurse -Exclude *jpg, *jpeg, *png, *pdf | ?{ $_.FullName -NotLike "C:\fullpath\.git\*" } | sls "<body>" | group path | select -unique name # exclude multiple sub folders dir ".\" -recurse -Exclude *jpg, *jpeg, *png, *pdf, *.mp4, *.ogv, *.webm, *pdf, *.png, *.svg, *.otf, *.eot, *.woff, *.ttf | ?{ $_.FullName -NotLike "C:\fullpath\.git\*" -and $_.FullName -NotLike "C:\fullpath\abc\*" } | sls "<body>" | group path | select -unique name # test dir without sls first to make sure you have the right "where" clause
38.12.6.5. Create script file
create C:\test.ps1 Run it
& "C:\test.ps1"
Pass parameters, always use param in the first line in the relevant scope
---- Begin script foo.ps1 ---- param([string]$foo = "foo", [string]$bar = "bar") Write-Host "Arg: $foo" Write-Host "Arg: $bar" ---- End script foo.ps1 ---- PS C:\> .\foo.ps1 -foo "foo" -bar "bar" Arg: foo Arg: bar
Function
function foo([string]$foo = "foo", [string]$bar = "bar")
{
Write-Host "Arg: $foo";
Write-Host "Arg: $bar";
}
function foo()
{
param([string]$foo = "foo", [string]$bar = "bar");
Write-Host "Arg: $foo";
Write-Host "Arg: $bar";
}
Scriptblocks
PS C:\>& {param([string]$foo = "foo", [string]$bar = "bar") Write-Host "Arg: $foo"; Write-Host "Arg: $bar"; }
Arg: foo
Arg: bar
38.12.6.6. Shared network drive
Sometimes cmd and powershell do not recognize the shared network drive
net use z: "\\remoteserver\subfolder"
38.12.6.7. Version
$PSVersiontable
38.12.6.8. Dracula Color Scheme
- https://github.com/dracula/powershell
Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process- In order to upgrade PowerShellGet
Install-Module -Name PowerShellGet -Force- (no term)
- Turns out have to lessen retristriction on policy permanently
38.12.7. Wireless hotspot
Tested with Win10
First, check if wireless adapter supports Hosted Network supported: Yes
netsh wlan show drivers # Create a wireless Hosted Network netsh wlan set hostednetwork mode=allow ssid=WindowsCentral key="yourkey" # Start the newly created Microsoft Hosted Virtual Adapter netsh wlan start hostednetwork
WinKey + x and select Network Connections, select any network adapter (LAN or wireless) currently have an internet connection, Properties > Sharing > Allow ohter network users to connect through this computer's Internet connection > Home networking connection select the newly created "Local Area Connection* 13 (or other numbers)"
Start and stop the Microsoft Hosted Virtual Adapter
netsh wlan stop hostednetwork netsh wlan start hostednetwork netsh wlan set hostednetwork mode=disallow netsh wlan set hostednetwork mode=allow
Change setting
netsh wlan set hostednetwork ssid=Your_New_SSID netsh wlan set hostednetwork key="abc" # View netsh wlan show hostednetwork netsh wlan show hostednetwork setting=security
It's not easy to delete the hostednetwork setting HKEY_LOCAL_MACHINE\system\currentcontrolset\services\wlansvc\parameters\hostednetworksettings Right-click the HostedNetworkSettings DWORD key, select Delete, and click Yes to confirm deletion. Reboot
38.12.8. SOCKS Proxy
Setup Windows to use SOCKS proxy for all internet connections. Win + s > Internet Options > Connections > LAN settings > Use a proxy server, go to Advanced, clear all protocols but config for SOCKS.
IP will be the proxy server IP but local DNS is used to resolve domain names to IP.
38.12.9. Windows Virtual PC
Install XP Mode on Windows 10 Pro and only Pro version can install
- Google Windows XP Mode and download the en-us version
- Enable Hyper-V in BIOS
- In Win 10, run optionalfeatures.exe to go to Windows Features, enable all Hyper-V. Restart Windows
- Control Panel > Admin Tools > Hyper-V manager
- Download 7-Zip. Choose 64-bit if Win is 64.
- Extract the downloaded XP mode file, go to folder Sources and find xpm file. right click and choose 7-zip > Open archive
- Find
VirtualXPVHD, extract it and rename toVirtualXPVHD.vhdfile - Open Hyper-V Manager, go to Virtual Switch Manage to create a Virtual Switch
- Type External, use default for others so that VM can use internet
- In Hyper-V Manager, select the one and only local virtualization server on the left pane.
- Action > New > Virtual Machine
- Specify a name
- Use Generation 1
- RAM should 512MB or 1024MB
- Select a Virutal Switch
- Select an existing Virtual Hard Disk
- Next, Finish
- On the right pane in Hyper-V Manager, click Connect then Start
- Set Region and Time
- License is required..
- Turn off Windows Update. Expect to see some driver installation failed
- Most likely, internet doesn't work. Turn off VM.
- Settings > Add Hardware > Legacy Network Adapter
- Choose the Virtual Switch, Apply. Start VM
38.12.10. Find physical path by shared path
On server, run net share
38.12.11. Process Monitor procmon.exe
To troubleshoot a process, e.g. OUTLOOK.EXE Filter > Filter, or Ctrl+L, add Process Name is not OUTLOOK.exe Exclude, Apply Tools > Count Occurrences > Column: Result, double click on each Result to filter events
38.12.12. Dosbox
http://www.dosbox.com/download.php?main=1
This is to run old DOS programs. Double click to install to C:\DOSBox Put all your old DOS programs to, e.g. C:\OLDGAMES
Then go to cmd, run C:\DOSBox\dosbox.exe
Z:\>MOUNT C C:\OLDGAMES Z:\>C: C:\>run your dos commands!
To automate the mount commands, double click on DOSBox-v-numberConfiguration, and a dosbox.conf text file pops up. Add these 2 under [autoexec]
[autoexec] MOUNT C C:\CT C:
38.12.13. Batch Script
F7 in command prompt to review command history for the current window
38.12.13.1. Setting
Usually at the top of a .bat file. @ in front makes the command apply to itself only @echo off :: turn off output of the command itself
38.12.13.2. Comment
Rem Your comment in one line
38.12.14. FART windows:FART
- Download
- Replace text in files
fart.exe -i -r "C:\Dir\To\Files\*.txt" original_text new_textfart.exe -i -r --remove *.txt "original text"- Say you want to remove double quotes. You will need to escape " with \"
fart -i -C --remove *.txt "\""- case insensitive
- whole word
- recursive
- Find and replace filename instead of contents
- Also find and replace in binary files
- preview. no action
I realize fart sometimes cannot find files.. Use Linux
38.12.15. Windows Credential Store Panel Windows Credential Store Panel
User Accounts > Your Account > Manage your credentials Or Credential Manager Instead of Web Credentials, choose Windows Credentials
OneNote needs a password to sync this notebook. Delete any records MicrosoftOffice{15|16}_Data:Live:cid=some set of numbers Which have no username and restart OneNote.
38.12.16. MAC address
getmac /v /fo list
38.12.17. Meter an Ethernet Connection
Run regedit: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\NetworkList\DefaultMediaCost
Right click on the key > Permissions > Advanced > Change next to Owner on the top
Type Administrators and Check Names
Check the box next to Replace owner on subcontainers and objects and click OK
Back in the Permissions for DefaultMediaCost window, click Administrators to select the group
Assign Full Control Allow, then OK
Right click DefaultMediaCost > Ethernet and Modify
Change from 1 to 2 for metered (DWORD 32-bit, Base Hexadecimal)
38.12.18. TS: "Installation Directory must be on a local hard drive"
Install .msi using admin privilege
Shift+Right Click on the msi file and select copy path
Open cmd run as admin and run: msiexec /i "the path"
38.12.19. TS: Win 10 random freezes
- Download latest drivers
- Win + R and type temp, delete all files
- Advanced Power Plan > Hard Drive > Never turn off
- BIOS turn off CPU C1E Function
- Adjust Virtual Memory for C drive
- Custom size: Initial size = Recommended, Max size = RAM * 1.5 in MB
- Uncheck
Automatically manage paging file size for all drives
- Win + R and mdsched.exe to do a memory check
- Check disk: right click on C drive > Properties > Tools > Check
- Win + S, cmd, run as admin,
sfc /scannow
38.13. Office 365
38.13.1. Exchange Online
38.13.1.1. Exchange Admin Center EAC ECP
- New View of EAC
admin.exchange.microsoft.comRecipients
- Mailboxes
- does not include distribution list
- UserMailbox, SharedMailbox
https://outlook.office.com/ecp/Recipients
- mailboxes
- User Mail Box only
- groups
- Distribution List, Dynamic Distribution List, Office 365
- Office 365 Group
- collaborate between users both insdie and outside your company. Shared workspace for conversations, files and calendar events, and a Planner
- shared
- Shared Mail Box
- ECP > permissions
admin roles
- Discovery Management
- Assigned roles
- Legal Hold
- Mailbox Import Export o365:role:mailbox import export
- Mailbox Search
- Members: add a person who should have the above roles
- Organization Management
- Assigned roles
- Mail Recipients o365:role:mail recipients
- ECP > compliance management
in-place eDiscovery & hold ms:ecp:in-place eDiscovery & hold
- Use the new and better ms:scc:content search
- Create a job with name, select all mailboxes or specific mailboxes, filter, set In-Place Hold settings, export to .pst file
- In-Place Hold: to hold/preserve items matching the search query for a specific of time so that users or automatic detention policy cannot delete them. Requires Exchange Online Plan 2 or Exchange Online Archiving license for each mailbox
38.13.1.2. OWA
- Outlook Web App. For O365 or Exchange Server 2016, it's called Outlook on the web
- Search
- Case insensitive
fromtoccbccparticipantsJerriFrye@contoso.com,JerriFrye, or"JerriFrye"can be usedfrom:"First Name Last Name"to:"First Name Last Name"
bodyandsubjectsubject:"Your Subject"- phrase Your Subject
subject:product plan- "product" or "plan" in the subject
subject:(product plan)- with both product and plan in the subject
ANDORfrom:"First Name Last Name" AND subject:"report"andfrom:"First Name Last Name"subject:"report"are equivalentfrom:"First Name Last Name" OR subject:"report"
-from:"Jerri Frye"sentreceived- only
MM/DD/YYYY sent:12/31/2019
- only
hasattachment:yeshasattachment:noisflagged:yesisflagged:no
- OWA Shared Mailbox
- Shared mailbox doesn't require a license and quota is 50gb each. If it exceeds the quota, it will be locked
- After it's added from Admin, user has to restart Outlook in order to see the shared mailbox
- It takes one hour for new members to see the new shared mailbox
- Open Shared Mailbox in OWA
- Click on your picture and select Open another mailbox
- Type the shared mailbox name
- Add Alias and Move Messages
- Add secondary email address to a shared mailbox and then setup Rules to that shared mailbox
- Logon to OWA and open the shared mailbox primary email address
- Gear > Options > Mail > Automatic processing > Inbox rules
- Say the primary email is: abc@abc.com and a secondary email is: xyz@abc.com
- You want to move emails sent to xyz@abc.com to a sub folder in the shared mailbox.
- You need to use
it includes these words in the message header = xyz@abc.com
- Add Members
- A new member by default has full access:
- Read, manage, and send as (send as the mailbox email address)
- A new member by default has full access:
38.13.1.3. Message trace
- Messages older than 90 days are unavailable. Filters are:
- sender
- recipient
- sender original IP address
- Url trace can only trace back 7 days
38.13.1.4. Microsoft Message Security & Compliance Center SCC ms:scc
- https://protection.office.com/
- Permissions: each child is called
Role group nameeDiscovery Manager: perform searches and place holds on mailboxes, SharePoint Online sites, and OneDrive for Business locations
- Roles
- Export
- RMS Decrypt
- Custodian
- Communication
- Review
- Preview
- Compliance Search
- Case Management
- Hold
- eDiscovery Administrator: assign users ms:scc:eDiscovery Administrator
- eDiscovery Manager: assign users
- Search
Content search ms:scc:content search
- Need to use Edge or IE to export search results
- New interface compared to the old ms:ecp:in-place eDiscovery & hold
- Assign yourself to ms:scc:eDiscovery Administrator
- It can include shared mailboxes, OneDrive and other O365 account/product types
- After creating a search, refresh, select and
More > Export resultsand export as .pst file
- O365 eDiscovery Export Tool needs to be installed
- Copy and paste the key to download
- Threat Management
Policy
- Anti-spam
- Default spam filter policy
- Spam and bulk actions
- threshold
- 7 (default)
- Allow lists
- Allow sender
- Allow domain
- Block lists
- Block sender
- Block domain
- Spam and bulk actions
- Connection filter policy
- IP Allow List
- IP Block List
38.13.1.5. Import PST Files
- Use ms:scc to import (network upload)
- https://docs.microsoft.com/en-us/microsoft-365/compliance/use-network-upload-to-import-pst-files
- A person who executes the import needs o365:role:mailbox import export and o365:role:mail recipients
- Import speed
- Several hours to upload 1 TB to Azure
- Import to O365 for at least 24 GB per day
- Multiple imports to different mailboxes are run in parallel
- Import limitation
- Any mailbox item larger than 150 MB will be skipped
- Can import to an inactive mailbox
- Ship them to Microsoft and ask them to import
38.13.1.6. LDAP
- Lightweight Directory Access Protocol
- In a network, a directory tells where in the network something is located
An LDAP directory:
- root directory
- Countries
Organizations
- Organizational Units: divisions, departments
- Individuals: people, files and shared resources e.g. printers
- An LDAP server is called a DSA (Directory System Agent)
- An LDAP directory can be distributed among many DSA's with replication and synchronization
- DNS (Domain Name System) is the directory system used to relate the domain name to a specific network address (each one is unique)
- But you have to know the domain name
- LDAP allows to search without knowing
- LDAP is the protocol that Exchange uses to communicate with Active Directory
38.13.2. Bookings
- Each Booking creates a user MyBookingName@contoso.com with Unlicensed
38.13.3. Teams
- Channel
Generalis created by default and members can add Conversation, Files or OneNote in a channel- Extra app can be added to a channel such as Stream, Website, Word, Excel etc.
- Mobile app for M$ Teams
- An org-wide team can be created and everyone will be added automatically
38.13.4. SharePoint
Site collection can have multiple subsites. Each site collection must have one top-level site. Defautl site collection and top-level site: yourcompany.sharepoint.com The default Team Site (yourcompany.sharepoint.com/TeamSite) is a top-site of the defautl site collection
See all site collections, Sharepoint admin center > site collections
Manage subsites and top-level site, go to one of the subsites and then click on Site Settings or go to the top-level site then Site Settings
38.13.4.1. Site Permissions
3 core default SharePoint permission groups: Owners, Members, Visitors.
You can add "Everyone except external" to a group.
Site Settings > People and Groups > Select the desired group > New
38.13.5. Office 2016
38.13.5.1. One-time-purchase Office License
Enter product key on a hard copy (card) on office.com/myaccount, get a digital product key.
To find out the digital product key of Office which is one-time-purchase.
Office 2016 32-bit on Win 64-bit cscript "C:\Program Files (x86)\Microsoft Office\Office16\OSPP.VBS" /dstatus
Office 2016 64-bit on Win 64-bit cscript "C:\Program Files\Microsoft Office\Office16\OSPP.VBS" /dstatus
Office 2013 32-bit on Win 64-bit cscript "C:\Program Files (x86)\Microsoft Office\Office15\OSPP.VBS" /dstatus
Office 2013 64-bit on Win 64-bit cscript "C:\Program Files\Microsoft Office\Office15\OSPP.VBS" /dstatus
38.13.5.2. Outlook
- Export Contacts as CSV and Reimport to Outlook
There's no option in Outlook for Mac to export contacts as CSV. OWA exports contacts without Company field. First, go to Outlook for Mac, select all contacts and drag them to a folder as VCF files. Open cmd on Windows, go to the folder with VCF files and run
copy /B *.vcf all_in_one.vcfUse vCard to LDIF/CSV Converter to convert all_in_one.vcf to CSV Open the csv in Excel and Save as CSV so that file uses CR+LF Go to OWA, delete all contacts Always use PC Outlook to import CSV. OWA barely functions. - Import Contacts
Import from .pst file handles the field mapping automatically. Import from .csv you would need to Clear Map and Default Map.
- Rules: Multiple Senders
Create a new contact folder and put contacts with emails. Create a rule which can be triggered when anyone in the contact folder belongs to "From" The rule is called "Sender is in specified Address Book"
- Rebuild OST
Make a copy of the .ost file before delete. Then open Outlook again. If you can't copy/delete .ost file, close Outlook, Skype for Business and Office applications.
- Rebuild .srs file
When rules don't work automatically but work manually, you should rename your-profile-name.srs at C:\Users\%username%\AppData\Roaming\Microsoft\Outlook
- Outlook for Mac
- Troubleshooting
Sorry, we're having server problems, so we can't add Office 365 SharePoint right now. Please try again later.
- Quit all Outlook and Office apps
- Spotlight search and open KeyChain Access
- Under Keychains > login on the right, delete all of these search results
- Exchange
- Office
- ADAL
- Launch Outlook
- Troubleshooting
38.13.5.3. Excel
38.13.5.4. OneNote
OneNote sync issue or remove crendentials. Refer to Windows Credential Store Panel
Move a OneOnte on SharePoint to personal OneDrive
- Export the whole Notebook to a local .onepkg file
- Open the .onepkg file and then File > Share > Move Notebook (you may need to Add a Place first)
- Ctrl + M
38.14. Cmder
- https://cmder.net/
- it's called Commander Console
- (no term)
- Fork of ConEmu
- (no term)
- Just executable no installation needed even for the full version
- Change the
cmder.exeto run as administrator
- Change the
- (no term)
- Preinstalled (full version)
- Git for Windows
38.14.1. Shortcuts
- Paste
S-InsertorC-vorright clickorC-S-v- Copy from Cmder
S-click- Traverse up in directory structure
C-A-u- Zoom in, Zoom out
C-Mouse Wheel- Preferences / Settings
Win-A-p- History search
C-r
38.14.1.1. Tab / console
- New tab
C-t- Close tab
C-w- Switch tab
C-TaborC-S-Tab
38.14.2. Settings
- Auto update is by default enabled
- Features > Colors
- Default Monokai
38.15. Reflector, ILSpy and Reflexil
C#, VB, F# and other .NET codes are compiled to Common Intermediate Language (CIL) or IL for short. Which is .dll file.
CIL is an object-oriented assembly language.
Red-gate .Net Reflector (trial) and ILSpy (open-source) can decompile a dll into source code.
Reflexil is a plugin for Reflect or ILSpy to change the OpCode for dll and export a new dll.
Reflexil v2.1 only works for ILSpy v2.4 but works for Reflector v9
Put Reflexil AIO (all-in-one version) dll to ILSpy.exe directory and run ILSpy.exe. View > Reflexil
Put Reflexil AIO dll to Reflector's Addins folder C:\Program Files (x86)\Red Gate\.NET Reflector\Desktop 9.0\Addins. Tools > Add-Ins, add the dll.
After edit, right click on the Object Browser > Reflexil > Save as
38.16. Project Management
38.16.1. Jira
38.16.1.1. Agile
- 4 Values
- individuals and interactions
- over processes and tools
- reduce multi-tasking
- working software
- over comprehensive documentation
- customer collaboration
- contract negotiation
- responding to change
- over following a plan
- 12 Principles
- Project manager's meeting responsibilities
- fulfills an observer role
- watches for unresolved issues
- watches for roadblocks
- ensures risks are decreasing over time
- communicates status to stakeholders
- Agile team size is less than 15 people
- scrum's name. Also Kanban board
- Swimlanes
- User Stories, To Do, Doing, Done
- Scrum vs Extreme Programming vs Kanban
Extreme programming
- Embracing change vs Inspect and adapt (in Scrum)
- User stories
- Continuous integration
- Test-driven development
Behavior-driven development
- Scenario format
- scenario
- a user story
- given
- some set of initial conditions
- when
- an event/action occurs
- then
- an outcome is expected
Gherkin and Cucumber.io
./features/card_minimum.feature# Comment @tag Feature: Card Minimum Scenario: Total charge is over the $2 credit card minimum Given Maria orders $3 of coffee from Li When Maria pays with a credit card Then Li should process the payment Scenario: Total charge is under the $2 credit card minimum Given Maria orders $1 of coffee from Li When Maria pays with a credit card Then Li should not process the payment
- 5 values
- Scenario format
- Communication
- Simplicity
- Feedback
- Courage
- Respect
Kanban
- Lean thinking
- Respect for people
- continuously improve
- new, backlog and incomplete
- each feature on an index card is to meet a business need
- <action> <result> -> achieve a business objective
- e.g. calculate tax for supplies ordered
- (no term)
- Group them by category and priority
- (no term)
- feature list reviewed > agreement by business > agreement by sponsor
- (no term)
- feature list is made and approved before each iteration starts
- (no term)
- estimate all features in terms of work at the beginning, and estimante relevant features before each iteration
- (no term)
- after the first estimate, create iteration, milestone and release plan
- (no term)
- In Speculate, decide which backlog or incomplete features should go in the current sprint
- (no term)
- Features (payment) > Themes (credit card payment) > Epics (visa payment) > Stories (actionable items.) > Agile estimation
- (no term)
Define a feature by using use case
- Actor (who) interacts with what or when what happens, actor does what
- Performance Requirement Card
- Feature ID: 123 - Invoices to Receive
- Performance criteria: 90 days will be the max allowed time to receive invoices
- Complexity factor: Medium
- Acceptability point of measurement: 90 day review of received invoices to be done by receiving manager (who to or what is used to measure)
- Verified by: Accounts receivable analyst
- Stages in Agile Life Cycle
- Envision > Speculate > Explore > Adapt (may go back to speculate) > Close
- Envision
what to build, resource and team you have, values & norms
- Project charter
- Scope, Objectives, Defined Stakeholders
- Describes the customer's visions of the final product and overall boundaries for the project
Sample: University of Waterloo
- Vision Statement
- "Help people enjoy the outdoors and learn more about the wonderful birds that live on our planet"
- Mission Statement
- "Creating a cutting-edge technology platform that combines a website, a smartphone application, and location awareness to help our customers identify local birds"
- Success criteria
- "Sign up 10,000 users within 3 months of initial version"
- Product Data Sheet (PDS)
- 1 to 3 pages
- project description and high level project scope, usually taken straight from project charter
- project objectives, the business value to provide
- Timeline
- Cost estimates
- environmental, safety, economic (budget), technical (meet standards), political, project schedule (certain people are only available in November), team or product development
- to balance (prioritize) scope, people, other resources, schedule and quality
- Set up collaboration tools
- e.g. work and meeting schedule, how they should cooperate
- actively listen to what others are saying
- attach the problem, not the person
- seek to understand first
- only focus on "this" sprint and feature
- if you see a problem, say something and make suggestions for resolution
- actively participate in the daily meetings
- be engaged
- solve conflict with the other pserson when possible
- if not achievable, both parties must come forward for help from management
- email is a low touch communication vehicle and should not be used to solve problems
- don't respond to text messages during meetings
- provide your full attention to the matter at hand
- be respectful of one another
- have the best interest of the project as your first priority
- share and respect the roles and responsibilities of each team member
- (no term)
Speculate, Explore and Adapt happen in each sprint or iteration
- sprint structure
- sprint size and keep it the same across the project
- In each sprint, 1 week for speculate and 1 week for adapt
- Determine sprint size by groups of features
- small (20 hours), medium 40 and large 80
- (no term)
- Speculate
- features based delivery plan and for each feature what estimates (time & resource) and risks are
- Requirements broken down to features. At the end of each sprint, they will be implemented
- if the sprint is 8 weeks, speculate phase is about 5 days
- (no term)
Explore
- Daily stand-up meetings. Each 15min or 30min max
- what they achieved, what they will achieve today, anything they need help with
- it's not the place to resolve issues
- honest peer reviews of features among business team, testing and technical team
- Identify roadblocks in daily work, collaboration, morale
- Adapt
- quick, often within one day
- final review of the features by the customer
- a documented meeting of team members to reflect on their performance (e.g. compare to plan, acknowledge member achievements)
- discuss what is and is not working (brainstorm ideas to resolve issues)
- agree to changes
- so that lessons are captured and shared and future sprint plans are reviewed and adjusted
- (no term)
- Close
- ensure all deliverables are completed
- Retrospective (share wins, what did we do well? what could improve)
- ensure vendors are paid and payments received
- (no term)
- Sprint Zero
- Finalize the vision
- purchase software
- contract vendors
- train
- decide on Sprint duration
- Release plan
- Envision > Speculate > Explore > Adapt (may go back to speculate) > Close
- Scrum
- 3 pillars
- Transparency
- Inspection
- Adaptation
- 5 Values
- Focus
- Respect
- Openness
- Courage
- Commitment
- Roles
- Product Owner
who define what will be built and the order. It's called Customer Representative in Extreme Programming
- Owns the product backlog, can cancel a sprint
- Product Backlog > Release Backlog > Sprint Backlog
- Requirements
- Enhancement Requests
- Defects
- User Stories
- New Features
- PO solictis estimates on backlog items from team
- 4-8 hours per week
- detailed, estimated (Planning Poker), emergent, prioritized
- Development team member
- 3 to 9 people
- self-organized
- a subset of product backlog
- (no term)
- Scrum master
- coach, encourage and facilitate self-management
- Events
- Sprint
It's part of the Scrum framework. each takes 30 days or less
- sprint planning
- entire scrum team is required
- first day of sprint
- planning all sprint work
- talk about new backlog items
- discuss all backlog items
write tasks for each story
- Tasking each story
- Tasks or steps to take to meet acceptance criteria (AC)
- Time-based estimate for each task
- Add stories into Sprint Backlog for this sprint
- Scrum master asks people to verbalize their commitments
- plan the coming sprint, plus 2 more sprints
- less than 2 hours or 8 hrs for a 30-day sprint
- Daily Scrum
- Sprint review ceremony, or Product Demo
- listen feedback from other stakeholders after a sprint ends
- less than 2 hrs or 4 hrs for 30-day sprint
- Sprint retrospective ceremony (Self-improvement)
- after sprint ends
- formal 2-hour meeting or 3 hrs for 30-day sprint
- Phases
- start
- create structure and safety
- data gathering
extract info from the team in an organized manner
- Starfish diagram
- Keeping doing, less of, more of, stop doing, start doing
- PANCAKE agenda
- Puzzles
- Appreciation
- News
- Challenges
- Aspirations
- Knowledge
- Endorsements
- insights
- generate ideas about how to improve the team based on the data gathered
- decisions
- choose clear action items
- SMART
- specific, measurable, achievable, relevant, time-boxed
- closing
- gather feedback on the retrospective for next time
- Artifacts
- Product Backlog
- Sprint Backlog
- Product Increment
- 3 pillars
38.16.1.2. Story
- In an agile framework, user stories are the smallest units of work
- User stories are sketched out by the product owner, and then the entire product team collectively determines detailed requirements
- As a {user role}, I want {goal} so that I {receive benefit/value}
- Acceptance criteria (AC) is on the back of the card
- AC primary perspectives
- client, developer, tester
- Acceptance criteria (AC) is on the back of the card
- Evaluate if user stories are valid or good
- I
- indenpendently valued
- N
- negotiate with Product Owner to figure out the value each story should deliver
- V
- Customer Valuable
- E
- Estimable
- S
- Small. A user story should be delivered within a sprint
- T
- Testable
- large grouping of work with common objective. How to split an epic into stories? Epic: I want to see a list of matching birds, so I can learn more about bird watching.
- F
- Flow. Story: As a bird finder, I want to see a list of matching birds on my smartphone, so I can learn more about birds when I'm away from my house
- E
- Effort
- Story: As a bird finder, I want to see a list of matching birds sorted by likely matches, so I can learn more about bird watching
- E
- Entry of data
- Story: As a bird finder, I want to see a list of matching birds based on my uploaded GPS coordinates, so I can learn more about bird watching
- D
- Data operations
- Story: As a bird finder, I want to see an editable list of matching birds, so I can learn more about bird watching
- B
- Business rules
- Story: As a bird finder, I want to see a list of matching birds based on the likelihood of a match on GPS coordinates and the image, so I can learn more about bird watching
- A
- Alternatives
- Story: As a bird finder, I want to see a list of matching birds located in my zip code, so I can learn more about bird watching
- C
- Complexity
- Story: As a bird finder, I want to see a list of matching birds based on color, GPS coordinates, and recorded autdio, so I can learn more about bird watching
- K
- Knowledge. Team has to do research and break it down into stories. Team create Story Spikes, they are not stories but the results of them can lead to stories
38.16.1.3. Feature
- As a Fitness Club Manager, I should be able to set up a new club member account
- It contains multiple workflows
38.16.1.4. Kanban board
- Board Setting > Setup Kanban backlog
- Assign status (To Do, In Progress, In Review, Done, etc.) to Kanban backlog
- Number of cards (issues) can be controlled for each column (In Progress, Done)
- Setup priorities of issues as swimlanes
- Reports: Control Chart, Cumulative Flow Diagram
38.16.1.5. Gadgets
A dashboard contains gadgets which show stats of different projects
- Average Age Chart
- how long issues were unresolved
- Sprint Health Gadget
- % time elapsed, % of work donw, % of scope change
- Jira Roadmap
- # of resolved vs # of unresolved
- Two Dimensional Filter Statistics
- x and y axises
- (no term)
- Assigned to Me
- Agile Wallboard
- a small version of the current sprint scrum board
- (no term)
- Pie Chart
- (no term)
- Sprint Burndown
- (no term)
- Days Remaining
38.16.1.6. Add-ons
- Settings > Add-ons > Find or Manage add-ons
- Gadgets
- Rich Text Gadget
Automation for Jira (paid)
- Project rules
- Close epic when all stories have been completed
- Close stories when epic is completed
- Planning Poker
Portfolio for Jira
- To link epics and tasks across projects as dependencies using link type. Default link type is Block
- e.g. Issue ABC-1 is blocked (as in Link other issues) by ABD-2 and ABE-3
- To link releases across projects as dependencies using Plan which groups selected projects
- Create cross-project release
38.16.2. YouTrack
38.16.2.1. Pricing
- Based on number of users. The more users, the less cost per user
- 1-3: free
- 3-100: USD $4.16/user
- 100-200: $3.33
- 200-300: $2.5
- 300-400+: $0.83
- Buy more users for more storage: 1 user = 1 gb
38.16.2.2. Project
- An issue can only have 1 project
- Try to link another issue in other project. Another issue needs to be created in the other project
38.16.2.3. Default fields
- State
- Settings > Custom Fields Settings > Fields List > State
- See which values are considered as Resolved, others are Unresolved
- Search
#resolvedor#unresolved
- Settings > Custom Fields Settings > Fields List > State
- Assignee
- May change to
Can specify multiple values - Search
- Refer to Login column in Users page for user name used in query
Assignee: li- include Li and Tom
- exclude Li
- exclude Li and Tom
- me or unassigned
- May change to
- Spent Time
- Require Time Tracking is enabled in project
- Search
Spent time: 1h .. 2h- greater than 1 day
- empty value
38.16.2.4. Attribute-based Search
- https://www.jetbrains.com/help/youtrack/incloud/attribute-based-search.html
- Check default fields for individual search query
projectproject: {Ad Ops}project: {Ad Ops} , Project-2
createdcreated: 2018-03-10 .. 2018-03-13- from a date to another date
created: * .. 2018-03-10- on or before a date
created: 2019-01-01 .. *- on or after a date
38.16.2.5. Reports
- Time Report
- https://www.jetbrains.com/help/youtrack/standalone/Time-Report.html
- Total amount of time spent
Prerequisites
- Enable time tracking in projects
- Estimate field
- e.g. Spent time
- Work Item Type
- e.g. Development, Testing, Documentation
Group by
- Project
- Work author
- per project
- Per
- e.g. per user (work author), issue or work item
- (no term)
- Issues per project
Useful reports
- Spent time percentage by project
- Projects
- include relevant projects
- Issue filter
created: 2018-11-01 .. *- Presentation
- Pie chart
- Show totals for
- Spent time
38.16.2.6. Mailbox Integration
- Filter messages by patterns
- Match message subject or text
38.16.3. Azure DevOps
38.17. Slack
38.17.1. Incoming Webhooks slack:incoming webhooks
- Create New App for one Workspace
- Activate Incoming Webhooks under Features on the left nav
- Add New Webhook to Workspace and select a channel
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXXwhich is user, channel specific- Test
curl -X POST -H 'Content-Type: application/json' -d '{"text": "hello world"}' http://.. - https://api.slack.com/incoming-webhooks#advanced_message_formatting
- attachment style
- https://api.slack.com/docs/message-attachments
38.17.2. Shortcuts
- List all shortcuts
⌘ /- Move focus to the next section
F6,S-F6⌃ ⌘ left/right
- Hover or select previous message in a channel
UporDown- Add a reaction to this message
C-S-\orCmd-S-\- Edit this message
e- Go to a conversation
Cmd-k
38.18. Image
38.18.1. Sample Image
38.18.1.1. Placeholder.com
http://via.placeholder.com/300x250/?text=hello+world
http://via.placeholder.com/300/ffffff/000000/?text=hello+world
Hex, first is background color and then text color
http://via.placeholder.com/300/ffffff/?text=hello+world
Auto text color
38.18.1.2. Text with Transparency
First color is background color and second color is font color whose default is 757575
https://imgplaceholder.com/420x320/transparent?text=hello https://imgplaceholder.com/420x320/transparent/fff?text=hello https://imgplaceholder.com/420x420/ca5353/757575?text=hello+world https://imgplaceholder.com/420x420/ca5353?text=hello+world https://imgplaceholder.com/420x420/ca5353?text=hello+world_br_Im+here+with+a+line+break https://imgplaceholder.com/420x420/ca5353?text=hello+world_br_Im+here+with+a+line+break&font-size=12 https://imgplaceholder.com/420x420/ca5353?text=hello+world&font-size=12&font-family=Roboto https://imgplaceholder.com/420x320/b13939/fff/glyphicon-tags https://imgplaceholder.com/420x320/transparent/fff/glyphicon-tags https://imgplaceholder.com/420x320/transparent/fff/ion-android-more-vertical https://imgplaceholder.com/420x320/b13939/fff/fa-android
38.18.1.3. Real Image - Picsum.photos
Pick a picture that you like :: https://picsum.photos/images https://picsum.photos/
Specific picture :: https://picsum.photos/200/300?image=0
Random picture :: https://picsum.photos/200/300/?random
Blur :: https://picsum.photos/200/300/?blur
Grayscale :: https://picsum.photos/g/200/300
Square :: https://picsum.photos/200
38.18.1.4. Free PNG with transparency
38.18.2. Convert image to favicon (.ico)
- https://converticon.com/
- https://realfavicongenerator.net/
- 16x16 32x32 64x64 (genesis) or any multiple of 2. WordPress recommends at least 512x512
- Best to use PNG to convert
38.18.3. WebP
Supported by Chrome desktop and android and Opera.
https://developers.google.com/speed/webp/faq#how_can_i_detect_browser_support_for_webp
Detect if request header accept literally contains image/webp. Browsers have these accept headers
*/* image/*
Some CDN support content negotiation in action means if the accept doesn't have image/webp then deliver .png or .jpg. How to deliver different cache based on the same URL? Nginx example
server { listen 80; server_name ei8gd7lwnymbl2d.cdnconnect.netdna-cdn.com jdorfman.cdnconnect.com; set $webp ""; if ($http_accept ~* image/webp) { set $webp "webp"; } location ~ /purge(/.*) { allow 192.168.0.1/24; deny all; proxy_cache_purge my_diskcached ei8gd7lwnymbl2d.cdnconnect10$myae$webp$1$is_args$args; } location / { [clipped] proxy_cache_key ei8gd7lwnymbl2d.cdnconnect10$myae$webp$uri$is_args$args; } }
Notice how the “$webp” variable gives us different cache keys for the various versions.
More on WebP and Nginx :: https://www.igvita.com/2013/05/01/deploying-webp-via-accept-content-negotiation/
38.18.4. Image Stock
38.19. cPanel WHM
38.19.1. WHM Auto Backup
Do it using WHM and don't do it using cPanel as it needs to setup scripts to do backup on cPanel search Backup Configuration > Enable https://www.inmotionhosting.com/support/product-guides/dedicated-hosting/setup-scheduled-cpanel-backups
Default backup directory is /backup
38.19.2. WHM/cPanel DNS Record Files
Directory /var/named/yourdomain.com.db
Create full backup and inside folder dnszones, there's yourdomain.com.db
38.19.3. WHM Add IP to Firewall - Permit SSH Access
In order to see the list of IPs, cat /etc/apf/allow_hosts.rules
38.19.4. WHM List of Accounts
Modify Account
- Privileges
- Shell Access (SSH)
38.19.5. WHM Security Advisor
38.19.6. WHM Access
https://example.com:2087 http://example.com/whm (port 2086) http://whm.example.com (port 80)
38.19.7. SSH access
For VPS account, you might need to add IP to allowed rule under "Add IP to Firewall".
38.19.8. autossl:whm
38.19.9. Remote MySQL connection - cPanel
Remote MySQL and add remote IP address. Use the cPanel domain for db host https://www.inmotionhosting.com/support/website/databases/setting-up-a-remote-mysql-connection-in-cpanel
38.19.10. Access Log - cPanel
Metrics > Raw Access. Current logs :: ~/access_logs Archived logs :: ~/logs
38.19.11. Root domain A record change - cPanel
You want to point the root domain to antoher server. Assuming root domain originally has A record 1.2.3.4 and new IP is 2.2.3.4
- Change root domain A record to 2.2.3.4
- Change mail.yourdomain.com CNAME yourdomain.com to A record 1.2.3.4
- Change smtp.yourdomain.com CNAME yourdomain.com to CNAME mail.yourdomain.com
- Change yourdomain.com MX yourdomain.com to mail.yourdomain.com
38.20. Web Hosting
38.20.1. Comparison
hosting:pantheon is good for hosting one website
Charged by monthly pageviews
| features | $25 | $100 | $400 |
| pageviews | 10k | 100k | 500k |
| storage gb | 5 | 20 | 30 |
| memory | 256 | 256 | 512 |
| PHP workers | 4 | 8 | 16 |
| database working set | 128 | 500 | 1gb |
hosting:wp-engine is good for 1, 5 and 15 websites
Charged by monthly visits and bandwidth. Multisite is possible.
| features | $35 | $115 | $290 |
| sites | 1 | 5 | 15 |
| visits | 25k | 100k | 400k |
| bandwidth gb | 50 | 200 | 400 |
| storage gb | 10 | 20 | 30 |
Digital Ocean has no bandwidth cap.
38.21. Tech Stack
- Software Tech Stack
- Siftery is acquired by G2 https://stack.g2.com/ tech stack:g2
- See what software/services other similar companies are using, and the pricing discount
- Siftery is acquired by G2 https://stack.g2.com/ tech stack:g2
- https://www.crunchbase.com/
- Web Technology Usage and Trends
- https://trends.builtwith.com/
- used by Mary Meeker from Kleiner Perkins in her Internet Trends Report
- (no term)
- https://w3techs.com/
- (no term)
- JavaScript
- https://bestofjs.org/
- https://npmtrends.com/
- https://bundlephobia.com/package/@emotion/react@11.11.1
- NPM bundle size and download time
- https://npm-compare.com/
- https://snyk.io/advisor/npm-package/react
- History of security
- Health
- Maintenance
- How often is the release?
- https://moiva.io/?npm=vite+webpack
- Compare NPM packages
- https://stateofjs.com
- (no term)
- http://hotframeworks.com/
- (no term)
- https://survey.stackoverflow.co/
38.22. Website Status
StatusCake - Free for 10 websites, 5 minutes check, random test locations Pingdom - Free for 1 website only.
38.23. Website Security, Testing
- Vulnerability testing
- Security google:site status
- https://sitecheck.sucuri.net/
- https://www.virustotal.com
- html:link:rel:noopener
- wp:ts:hacked
- https://securityheaders.com/
- SpeedCurve and Calibre">Monitor web performance budgets using chrome:lighthouse and google:pagespeed
38.23.1. XSS
Tests
http://a.ca/abc-xyz--><form><button formaction='/esi.attack.sim'><!-- http://a.ca/abc-xyz--><script src='a' onerror=alert(document.domain)>
38.24. Webhook
- Webhook.site
- ngrok.com
- Host your local http app on the internet
- See 17.17.27
38.25. Send password
38.26. RESTful API
38.27. Screencast + Audio, FFmpeg
38.27.1. ShareX
ShareX Download the portable version, Task Settings > Capture > Screen recorder > Screen recording options… Click Download to download the latest FFmpeg.exe Under Sources, hit Refresh until you see the right Audio source. Then go back to UI, Capture > Screen recording
38.27.2. OBS Studio linux:app:obs-studio
38.27.4. ffmpeg
- Convert a gif to mp4
-pix_fmt yuv420p- MP4 store pixels in different formats and this is the max compatible across all browsers
-vfMP4 videos using H.264 need to have a dimensions that are divisible by 2
ffmpeg.exe -i animated.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" video.mp4<img src="video.gif" alt="" width="400" height="300" /> <video autoplay="autoplay" loop="loop" width="400" height="300"> <source src="video.mp4" type="video/mp4" /> <img src="video.gif" width="400" height="300" /> </video>
- Refer to youtube:encoding
38.28. Sketch Diagrams
38.29. Video
38.29.1. JW Player video:jw
- Started as an open source used by YouTube before it's acquired by Google. Now ti's proprietary and used by ESPN, Electronic Arts and AT&T
- Pricing
- Custom Enterprise pricing, bills every 3 months
38.29.2. Brightcove video:brightcove
- Public company
- Pricing
- Custom pricing
- Migration
- Looks like there's no way to migrate a sub account (its videos, playlists, players, stats) to another master account
38.29.3. Telaria
- Video Management Platform (VMP)
- Video advertising
- previously called SlimCut
- Features
https://outstream.telaria.com/
- Formats
- Carousel, Image + Copy, Vertical Video, Outstream, Video + Copy, Leave Behind, Footer Banner
- (no term)
Cost
- Direct sales on desktop/tablet
- CAD $2.5 CPM on the first 4M impressions monthly
- CAD $1.5 CPM in between 4M and 15M impressions monthly
- CAD $1 CPM for over 15M impressions monthly
- Direct sales on mobile
- Addtional CAD $0.5 CPM added on top of desktop/tablet
- Third party tags on desktop/tablet
- CAD $2.5 CPM on the first 4M imp monthly
- CAD $1.5 CPM in between 4M and 15M imp monthly
- CAD $1 CPM for over 15M imp monthly
- Third party tags on mobile
- Addtional CAD $0.5 CPM added on top of desktop/tablet
38.29.4. YouTube Download
38.29.5. 猫抓
- [[- Chrome Extension https://chrome.google.com/webstore/detail/%E7%8C%AB%E6%8A%93/jfedfbgedapdagkghmgibemcoggfppbb/related?hl=zh-CN
][Chrome Extension]]
- Get the
m3u8link- Method 1: Play the video and search
m3u8in Network on Chrome - Method 2: Open 猫抓, grab the link
- Method 1: Play the video and search
- Open 猫抓 > 媒体控制/其他功能 > M3U8解析器 > paste the link > 解析 > 调m3u8DL下载 > 合并下载
38.29.6. Kodi
38.29.6.1. Settings
- Display Chinese characters, you need to go to
Settings > Appearance > Fonts > Arial based
- Add apps to Homepage e.g. Video
Settings > Appearance > Settings > Add-on
38.29.6.2. File Manager
- https://aznhusband.github.io/ App: icdrama
- http://fusion.tvaddons.ag/
Install from zip file Folder xbmc-repos > english >
- repository.exodus-1.0.1.zip (Exodus repository)
- repository.xbmchub-1.0.6.zip (TVADDONS.ag Addon Repository)
- chinese-repository.kodi-addons-chinese-1.0.2.zip (Kodi Add-ons of Chinese)
- chinese-repository.kodi-repo.zip (Joname's repo)
- chinese-repository.xbmc-addons-chinese-1.2.0.zip (Chinese Add-ons repository)
Install from zip file Folder isengard > all > superrepo.kodi.isengard.all-0.7.04.zip (Superrepo All repository)
38.29.6.3. Addons
Video Exodus (Exodus Artwork) Use Torba, Putlocker as the video sources (Exodus repository) Phoenix (TVADDONS.ag Addon Repository) icdrama.se (Superrepo All repository) azdrama.net (Superrepo All repository)
Subtitle 163sub (Chinese Add-ons repository) Sub HD (Chinese Add-ons repository) subom (Chinese Add-ons repository)
zimuku (Chinese Add-ons repository)
38.30. Audio
38.30.1. SoundCloud audio:soundcloud
- Plan
- SoundCloud Basic (Free)
- 3h upload time
- SoundCloud Pro Unlimited (CAD $15/month)
- Unlimited upload time
- Scheduled releases
- Limit for all plans
- single sound file < 4GB and time per track < 6hr45min
- SoundCloud Basic (Free)
- Migration
- Cannot merge accounts
- One email address one SoundCloud account
38.30.2. LibSyn
- Public company
- Features
- IAB v2.0 certified stats: user agent, geographic, download destinations (different podcast platforms), social media, day/week/month, date range, episode breakdown, report downloading
- Publish to podcast platforms using Libsyn RSS Feed and OnPublish
- Apple Podcast
- Spotify
- Google Podcasts
- Pandora
- WordPress Libsyn Publisher Hub
- Publish Audio, Video, PDF and Text Posts
38.31. Google
38.31.1. Google Public DNS
8.8.8.8 8.8.4.4
38.31.2. Google S2
Get favicon of any website http://www.google.com/s2/favicons?domain=http://superuser.com/
38.31.3. Google reCaptcha
- https://developers.google.com/recaptcha/intro
- allows you to verify if an interaction is legitimate without any user interaction. It is a pure JavaScript API returning a score, giving you the ability to take action in the context of your site: for instance requiring additional factors of authentication, sending a post to moderation, or throttling bots that may be scraping content
- An owner can remove another owner
38.31.3.1. Client - Basic
<html> <head> <title>reCAPTCHA demo: Simple page</title> <script src="https://www.google.com/recaptcha/api.js" async defer></script> </head> <body> <form action="?" method="POST"> <!-- has to have class="g-recaptcha" --> <div class="g-recaptcha" data-sitekey="your_site_key"></div> <br/> <input type="submit" value="Submit"> </form> </body> </html>
38.31.3.2. Client - Defer loading, explict rendering one widget/instance
<script type="text/javascript"> var onloadCallback = function() { alert("grecaptcha is ready!"); grecaptcha.render('html_element', { 'sitekey' : 'your_site_key' }); }; </script> <form action="?" method="POST"> <div id="html_element"></div> <br> <input type="submit" value="Submit"> </form> <script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit" async defer> </script>
- Use namespace
// v2 2020-10-22 myNamespace = { myRender: function() { grecaptcha.render('html_element', { 'sitekey' : '...' }); }, loadReCaptcha: function() { var g = document.createElement('script'), s = document.scripts[0]; g.src = 'https://www.google.com/recaptcha/api.js?render=explicit'; // cannot use dots in onload e.g. myNamespace.myRender // https://www.google.com/recaptcha/api.js?onload=myNamespace.myRender&render=explicit g.async = true; g.defer = true; g.onload = function() { // this hack does not guarantee grecaptcha.render is loaded // have to wrap with grecaptcha.ready grecaptcha.ready(function() { grecaptcha.render('html_element', { 'sitekey' : '...' }); }); }; s.parentNode.insertBefore(g, s); }, }; myNamespace.loadReCaptcha();
38.31.3.3. Client - Multiple reCaptcha widgets/instance
<html> <head> <title>reCAPTCHA demo: Explicit render for multiple widgets</title> <script type="text/javascript"> var verifyCallback = function(response) { alert(response); }; var widgetId1; var widgetId2; var onloadCallback = function() { // Renders the HTML element with id 'example1' as a reCAPTCHA widget. // The id of the reCAPTCHA widget is assigned to 'widgetId1'. widgetId1 = grecaptcha.render('example1', { 'sitekey' : 'your_site_key', 'theme' : 'light' }); widgetId2 = grecaptcha.render(document.getElementById('example2'), { 'sitekey' : 'your_site_key' }); grecaptcha.render('example3', { 'sitekey' : 'your_site_key', 'callback' : verifyCallback, 'theme' : 'dark' }); }; </script> </head> <body> <!-- The g-recaptcha-response string displays in an alert message upon submit. --> <form action="javascript:alert(grecaptcha.getResponse(widgetId1));"> <div id="example1"></div> <br> <input type="submit" value="getResponse"> </form> <br> <!-- Resets reCAPTCHA widgetId2 upon submit. --> <form action="javascript:grecaptcha.reset(widgetId2);"> <div id="example2"></div> <br> <input type="submit" value="reset"> </form> <br> <!-- POSTs back to the page's URL upon submit with a g-recaptcha-response POST parameter. --> <form action="?" method="POST"> <div id="example3"></div> <br> <input type="submit" value="Submit"> </form> <script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit" async defer> </script> </body> </html>
38.31.3.4. api.js options
| onload | Optional. The name of your callback function to be executed once all the dependencies have loaded. | |
| render | explict onload | Optional. Whether to render the widget explicitly. |
| Defaults to onload, which will render the widget in the first g-recaptcha tag it finds. | ||
| hl | See language codes | Optional. Forces the widget to render in a specific language. |
| Auto-detects the user's language if unspecified. |
38.31.3.5. class="g-recaptcha" tag attributes
| tag attribute | grecaptcha.render parameter | Value | Default | Description |
| data-sitekey | sitekey | Your sitekey. | ||
| data-theme | theme | dark | light | Optional. Color |
| light | ||||
| data-type | type | audio | image | Optional. |
| image | ||||
| data-size | size | compact | normal | Optional. |
| normal | ||||
| data-tabindex | tabindex | 0 | Optional. | |
| data-callback | callback | Optional. | ||
| To execute when the user submits | ||||
| a successful response: g-recaptcha-response | ||||
| data-expired-callback | expired-callback | Optional. | ||
| To execute when recaptcha response expires | ||||
| and the user needs to solve a new CAPTCHA |
38.31.3.6. JavaScript API
- grecaptcha.render(container,parameters)
Renders the container as a reCAPTCHA widget and returns the ID of the newly created widget. container :: The HTML element to render the reCAPTCHA widget. Specify either the ID of the container (string) or the DOM element itself. parameters :: An object containing parameters as key=value pairs, for example, {"sitekey": "your_site_key", "theme": "light"}. See grecaptcha.render parameters.
- grecaptcha.reset(opt_widget_id)
Resets the reCAPTCHA widget. opt_widget_id Optional widget ID, defaults to the first widget created if unspecified.
- grecaptcha.getResponse(opt_widget_id)
Gets the response for the reCAPTCHA widget. opt_widget_id Optional widget ID, defaults to the first widget created if unspecified.
38.31.3.7. Server Validation
- For web users, a new field (g-recaptcha-response) will be populated in HTML and you can get the user’s response in one of three ways
- A new form field with id
g-recaptcha-responseis created inside the container- POST parameter when the user submits the form on your site
grecaptcha.getResponse(opt_widget_id)after the user completes the CAPTCHA challenge- As a string argument to your callback function if
data-callbackis specified in either the g-recaptcha tag attribute or the callback parameter in thegrecaptcha.rendermethod
- A new form field with id
- URL: https://www.google.com/recaptcha/api/siteverify
- METHOD: POST
- Parameters
- secret
- Required. The shared key between your site and reCAPTCHA
- response
- Required. The user response token provided by reCAPTCHA, verifying the user on your site
- remoteip
- Optional. The user's IP address
API Response
{
"success": true|false,
"challenge_ts": timestamp, // timestamp of the challenge load (ISO format yyyy-MM-dd'T'HH:mm:ssZZ)
"hostname": string, // the hostname of the site where the reCAPTCHA was solved
"error-codes": [...] // optional
}
Error codes missing-input-secret :: The secret parameter is missing. invalid-input-secret :: The secret parameter is invalid or malformed. missing-input-response :: The response parameter is missing. invalid-input-response :: The response parameter is invalid or malformed. bad-request :: The request is invalid or malformed.
private function validateReCaptcha() { if (!isset($_POST['g-recaptcha-response'])) return FALSE; $C_CAPTCHA = $_POST['g-recaptcha-response']; $C_CAPTCHA_google_request = [ 'url' => 'https://www.google.com/recaptcha/api/siteverify', 'options' => [ 'secret' => 'your-key', 'response' => $C_CAPTCHA, // 'remoteip' => $_SERVER['REMOTE_ADDR'], // optional ], ]; $ch = curl_init(); $options = [ CURLOPT_URL => $C_CAPTCHA_google_request['url'], CURLOPT_POSTFIELDS => http_build_query($C_CAPTCHA_google_request['options']), CURLOPT_FOLLOWLOCATION => 1, // CURLOPT_HEADER => 0, // not to include header in response CURLOPT_RETURNTRANSFER => 1, // CURLOPT_SSL_VERIFYPEER => 0, // CURLOPT_SSL_VERIFYHOST => 0, // CURLOPT_POST => 1, ]; curl_setopt_array($ch, $options); $C_CAPTCHA_google_result = json_decode(curl_exec($ch), TRUE); curl_close($ch); if (is_array($C_CAPTCHA_google_result) && $C_CAPTCHA_google_result['success'] === TRUE) { return TRUE; } }
38.31.3.8. CSS
/* center v2 2020-10-22 */ .g-recaptcha > div {margin:0 auto;} /* The button might be too large for width < 320px */ @media (max-width: 320px) { .g-recaptcha { transform:scale(0.77); -webkit-transform:scale(0.77); transform-origin:0 0; -webkit-transform-origin:0 0; } }
38.31.4. Google Search - Advanced
https://bynd.com/news-ideas/google-advanced-search-comprehensive-list-google-search-operators/
https://support.google.com/websearch/answer/2466433
Search operators: google:search operators
site:nytimes.com site:.gov @twitter camera $400 #throwbackthursday jaguar speed -car "tallest building" largest * in the world $50..$100 marathon OR race related:time.com # get details about a site info:time.com cache:time.com
Google Search usually ignores punctuation that isn’t part of a search operator.
38.31.5. Google Ads Personalization Setting
38.31.6. Google Partners
First, set up your company as a Google Partners.
- Use a non-gmail email address to create an account to google.com/partners e.g. abc@yourcompany.com
- abc@yourcompany.com has to have edit or admin access to an AdWords account. This AdWords account can be registered using any Google account. Login to that AdWords account and add abc@yourcompany.com with edit or admin access.
- Once you created one Partner Profile account, go to Overview > My Profile, fill up address and phone number and select AdWords account as the "Top-level AdWords manager account"
- Go back to Overview, and add your company
After that, any Google account users without @yourcompany.com can affiliate with that company. However, an individual can only affiliate with one company.
- Sign in as abc@yourcompany.com to Google Partners and go to Overview > My Company > People to approve affiliation requests.
An Individual can work with who ever they wish, but can only affiliate their credentials of passing the required exams to ONE Company.
- An individual is now required to have admin, standard or read-only access to the company's AdWords manager account.
Google Partner is set up in basically for Companies that meet certain requirements of Certification, Ad Spend, and Best Practices qualify and earn the GOOGLE PARTNERS BADGE.
The benefit is to the company to have your credential of passing the exams on their roster, or towards them fulfilling the TEST requirements to become a Google Partner
Individuals who receive certification but are not considered Google Partners unless they meet the other requirements of Best Practices (from their linked MCC account) and Ad Spend levels.
38.31.7. Gmail
38.31.7.1. Use Gmail SMTP to send emails
- Turn off 2-Step-Verification or use the App Password.
- Turn off 2-Step-Verification
- If you can't setup App Password, it's because 2-Step-Verification is not turned on
- Let less secure apps use your account
- Which is here: https://myaccount.google.com/lesssecureapps
- Turn off 2-Step-Verification
38.31.7.2. Non-gmail email address as Google Account
- First create a Gmail account abc@gmail.com then go to My Account > Personal Info & privacy > Your personal info > Email > Advanced > Alternate emails > Add alternate email
- This is to add an alternate non-gmail email address to your Gmail email account
- These alternate non-gmail email addresses can be used to signin, recover password, get notifications and more
- Using Google products e.g. Google Docs, will show your primary gmail email address
38.31.7.3. Bookmark to go directly to a Gmail account
38.31.7.4. Filter and Search Operators
- https://support.google.com/mail/answer/7190?hl=en
- Search Operators are used inside the search box
from:(a@a.com) to:(b@b.com) subject:(Update Changes #*)ORor{}from:(a@a.com OR b@b.com)from:a@a.com OR from:b@b.com{from:a@a.com from:b@b.com}from:(a@a.com OR b@b.com) ("4326737" OR "898-849-9531")
- has:nouserlabels
- has:userlabels
is:unreadis:readcategory:updates-e.g. Not starred-is:starred
38.31.7.5. Shortcuts
- Action
- Toggle star
s
- Navigation
- Go to threadlist, go up
u
38.31.8. Google Sheets
38.31.8.1. Shortcuts
- Menu
- Help > Keyboard Shortcuts or
C-/- Enable compatible shortcuts
- Search a shortcut
- (no term)
- Insert row/columns or Open insert menu
C-M-=orC-M-S-=
- Insert current date time
⇧ ⌘ ;
38.31.9. Google Maps
- Embed single location
- https://www.embedgooglemap.net/
- (no term)
- Embed multiple locations, use My Maps of Google Maps
38.31.10. Google Correlate
Use it to find other search keywords that are related to your keyword!
38.31.11. Google Planning Tools
38.31.12. Tools for Web Developers
38.31.12.1. Lighthouse in DevTools chrome:lighthouse
Refer to npm:lighthouse DevTools > Audits tab (among Elements, Console, etc.)
- View and compare json report files
- https://googlechrome.github.io/lighthouse/viewer/
- Save as Gist
- save report to a secret Gist under your GitHub account. The viewer link with Gist is publicly available
38.31.12.3. PageSpeed google:pagespeed
38.31.12.4. Fetch as Google google:search console
- For sites that you don't own
- https://technicalseo.com/seo-tools/fetch-render/ or do an iframe like this and then use Fetch as Google
<html> <head><title>Fetch & Render Proxy</title> </head> <body style="margin:0px;padding:0px"> <iframe src="http://www.example.com/" frameborder="0" style="overflow:hidden;height:10000;width:1024" height="10000" width="1024"></iframe> </script> </body> </html>
38.31.12.5. Rich Snippet Testing Tool, Structured Data Testing Tool google:json-ld
Test websites that you own and don't
38.31.12.6. Google Site Status (security) google:site status
Google Site Status :: https://transparencyreport.google.com/safe-browsing/search
38.31.12.7. Install with headless chromium
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - sudo apt install -y nodejs # install chromium browser sudo apt install chromium-browser # or # sudo apt-get install chromium npm i -g lighthouse lighthouse --chrome-flags="--headless" https://github.com
38.31.13. Google Experts
38.32. Translation
tool:Poedit Open .pot and .po files in Poedit
Online Editor. Send .po file to customer who will translate online and they can save and send back the translated .po!
38.33. HTML Minimizer, document.write
Online minimizer
- http://www.willpeavy.com/minifier/
- https://www.textfixer.com/html/compress-html-compression.php
- http://htmlcompressor.com/compressor/
Node.js minimizer
tool:html:document.write Online convert a single line html to document.write
- http://accessify.com/tools-and-wizards/developer-tools/html-javascript-convertor/
- http://www.andrewdavidson.com/convert-html-to-javascript/
document.write("string") where string has to escape:
- double quotes
\" - backward slash
\\ - forward slash
\/as "</tag>" causes an error
Use document.write is not a good idea and the usage will be blocked only when all the following conditions are met:
- The user is experiencing a very poor network connectivity,
- The script is parser-blocking (neither async nor defer attributes) and is not already in the browser cache,
- The instruction is added in the top level document (e.g. iframes won’t be concerned),
Use e.innerHTML('') to insert elements, refer to js:external script to load javascript
38.34. Data file validator and visualizer
38.35. File Transfer
- https://send.firefox.com
- up to 2.5GB
- (no term)
- WeTransfer
- Free
- file size send limit 2GB
- Pro
- $12/month
- 1 TB
- 20 GB one file
- Your own branding page with custom domain and URL
38.36. CloudConvert.com
- Convert any file to another file format
- Provides NPM module for batch processing
38.37. Online App Publishing
38.37.1. https://glitch.com
- Node.js with SQLite, Webpage
- Projects created while not signed in will be deleted after 5 days
- Change project name e.g. abc-xyz
- Public access
https://glitch.com/~abc-xyz- Edit project file at
public/index.html https://glitch.com/edit/#!/abc-xyz?path=public/index.html:1:0
- anonymous viewers can't see it. When remixing, it's cleared and it's not copied
.datafolder to store data. When remixing, it isn't copied- Refer to vs:code:ext:glitch.glitch
- Project Setting
- Turn off
Refresh on app changeotherwise it always refresh for the root project url
- Turn off
- run
refreshin console
38.37.2. https://www.bitballoon.com
Webpage
38.37.3. https://zeit.co/
Node.js, Dockerfile
38.38. Email Service and Email Marketing
38.38.1. SendGrid
- Create a user per website who can use to authenticate SMTP and API
- Sign to main account and go to Settings > Credentials
- Add a header
X-SMTPAPIas JSON to assign category shown on SendGrid.com Stats and Activity- Drupal way
$message['header']['X-SMTPAPI'] = '{"category": ["fromwebsiteA"]}'
- Drupal way
- Drupal
- Use the SMTP Authentication Support module (smtp) and go to
/admin/config/system/smtpto config - Install Options - turn this module on
- SMTP server: smtp.sendgrid.net
- SMTP port: 465 (587 is being used…)
- Use encripted protocol: Use SSL
- Use the SMTP Authentication Support module (smtp) and go to
- Wordpress
- Install plugin SendGrid Refer to GitHub for setting config in wp-config.php
38.38.1.1. Whitelabel
- Domain whitelabel
- adds a SPF (Sender Policy Framework) record (dns:spf) to your domain and DKIM (DomainKeys Indentified Mail) entries, which authorizes and authenticates SendGrid sending on behalf of your domain. It basically removes "sent on behalf of SendGrid" or "via" message that some email providers dispaly
- The sent domain is the domain of the email address from "From"
- Usually, you need to create one subdomain per main domain. e.g. delivery.yourdomain.com
- With Automated Security checked, SendGrid will show you to create 3 CNAME records for your DNS registrar
- Some DNS registrar might not support
_in subdomains in CNAME host. In this case, don't check Automated Security - Without Automated Security checked, 1 MX and 2 TXT records are provided from SendGrid
- Email link whitelabel
- adds a CNAME record for a a subdomain that you choose, which masks click and open-tracking links to your domain rather than a SendGrid domain. e.g. In email body there's a link yourdomain.com. Before email link whitelabel is setup, the sent out email with this link changed to SendGrid domain. After whitelabel is setup, this link is changed to e.g. click.yourdomain.com
- This subdomain has to be different from the Domain whitelable subdomain
- SendGrid will show you 2 CNAMES per subdomain setup
38.38.1.2. PHPMailer
date_default_timezone_set('America/Toronto'); require_once(__DIR__.'/../../../wp-includes/class-phpmailer.php'); $receipients = array('a@b.com'); $subject = ''; $body = ''; $html = <<<HTML <html> <head> <title>{$subject}</title> </head> <body> {$body} </body> </html> HTML; $mail = new \PHPMailer(); $mail->isSMTP(); //$mail->SMTPDebug = 2; // 0 for off in production $mail->SMTPDebug = 0; $mail->Host = 'smtp.sendgrid.net'; $mail->SMTPAuth=true; $mail->Username= 'apikey'; // literally 'apikey' $mail->Password= 'secret key'; $mail->Port = 587; $mail->CharSet= 'UTF-8'; $mail->SetFrom('donotreply@myweb.com', 'My Website'); $mail->addCustomHeader('X-SMTPAPI', '{"category": ["fromwebsiteA"]}'); foreach ($reciepients as $v) { $mail->AddAddress($v); } $mail->Subject = $subject; $mail->MsgHTML($html); if (!$mail->Send()) { return 'error: '.$mail->ErrorInfo; } else { return 'success'; }
38.38.1.3. cURL sends emails
# basic curl --request POST \ --url https://api.sendgrid.com/v3/mail/send \ --header 'Authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{"personalizations": [{"to": [{"email": "recipient@example.com"}]}],"from": {"email": "sendeexampexample@example.com"},"subject": "Hello, World!","content": [{"type": "text/plain", "value": "Heya!"}]}' # send to multiple receipients curl --request POST \ --url https://api.sendgrid.com/v3/mail/send \ --header 'authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{"personalizations": [{"to": [{"email": "recipient@example.com"}],"cc": [{"email":"recipient2@example.com"}, {"email": "recipient3@example.com"}, {"email":"recipient4@example.com"}]}], "from": {"email": "sendeexampexample@example.com"},"subject":"Hello, World!", "content": [{"type": "text/plain", "value": "Heya!"}]}' # scheduled curl --request POST \ --url https://api.sendgrid.com/v3/mail/send \ --header 'authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{"personalizations": [{"to": [{"email": "recipient@example.com"}]}],"from": {"email": "sendeexampexample@example.com"},"subject":"Hello, World!","content": [{"type": "text/plain","value": "Heya!"}], "send_at" : UNIX_TIMESTAMP_HERE}'
38.38.2. Dialog Insight
- HQ in Montreal
- Clients
- Desjardins, L'Occitane, Estée Lauder, Metro, Transat, TD, Cascades, La Presse, CAA, Cossette and Ricardo Media
- Web Services, API
38.38.3. DMARC, SPF, DKIM dns:dmarc
- DMARC
- Domain-based Message Authentication, Reporting, and Conformance
- (no term)
- DMARC serves as a method of authentication for your brand, alongside Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM)
- (no term)
- What sets DMARC apart from the other two authentication methods is its reporting capability
- (no term)
- DMARC is an email authentication technology that protects a domain from being used in phishing and spoofing attempts by using a signing policy to define how receiving inbox providers should handle messages that fail an authentication check. DMARC also allows for a reporting mechanism in which inbox providers can send reports on email that appears to be sent from a certain domain back to the domain owner
- (no term)
- Inbox providers that support DMARC will attempt to validate both DKIM and SPF, and depending on the outcome of those checks, will look to the sending domain's DMARC policy on how to handle emails that fail authentication
- (no term)
- If the inbox provider is able to successfully validate either DKIM or SPF, the email will continue on its normal path. If the message fails both DKIM and SPF authentication checks, however, the inbox provider will enforce the sender’s DMARC policy, which specifies how the email should be handled if it fails authentication and where to send any reports
- (no term)
- DMARC has three levels of policy
- None
- If the policy is set to none, the receiving inbox provider monitors and reports on metrics
- Quarantine
- If the DMARC policy is set to quarantine, the inbox provider places messages that fail the required checks into the user's spam folder
- Reject
- If the policy is set to reject, the inbox provider will block any message that fails authentication
- (no term)
- https://sendgrid.com/blog/dmarc-domain-based-message-authentication-reporting-conformance/
- DMARC Specifications on DNS Tags and Values
- https://dmarc.org/resources/specification/
TXT _dmarc v=DMARC1; p=none; rua=mailto:dmarc@myweb.ca; ruf=mailto:dmarc@myweb.ca; fo=1; adkim=r; aspf=r;Sample
- v
- must have value
DMARC1and should be the first tag - p (required)
- Policy applies to the domain queried and to subdomains, unless subdomain policy is explicitly described using the
sptag. Possible values.
- none
- Domain Owner requests no specific action be taken
- quarantine
- Domain Owner wishes to have email that fails the DMARC check be treated by Mail Receivers as suspicious. This could mean "place into spam folder", "scrutinize with addtional intensity", and/or "flag as suspicious"
- reject
- Domain Owner wishes for Mail Receivers to reject email that fails the DMARC check. Rejection should occur during the SMTP transaction
- adkim
- default r. Indicates whether strict or relaxed DKIM Identifier Alignment mode is required by the Domain Owner.
sfor strict mode - aspf
- same as adkim but for SPF Identifier Alignment mode
- fo
- default 0. Failure reporting options for generation of failure reports. This tag is ignored if
ruftag is not specified. Colon-separated of the below values
- 0
- Generate a DMARC failure report if all underlying auth mechanisms fail to product an aligned "pass" result
- 1
- Generate a DMARC report if any underlying
- d
- Generate a DKIM report if the message had a signautre that failed evaluation, regarless of its alignment
- s
- Generate an SPF report if the message failed SPF evaluation, regardless of its alignment
- rua
- Addresses to which aggregate feedback is to be sent. Comma-separated
- ruf
- Addresses to which message-specific failure info is to be reported. Comma-separated
- pct
- default 100. Percentage of messages from the Domain Owner's mail stream to which the DMARC policy is to be applied
- sp
- Requested Mail Receiver policy for all subdomains. Indicates the policy to be enacted by the Receiver at the request of the Domain Owner. It applies only to subdomains of the domain queried and not to the domain itself. If absent, the policy specified by the p tag must be applied
- (no term)
- Refer to dns:dkim and dns:spf
- (no term)
- For adding another subdomain as in
From: e.mydomain.com- Add a SPF record that refers to the root domain's SPF record
TXT e v=spf1 include:mydomain.com -all- Add a DKIM record (may just duplicate the root domain's DKIM record)
TXT customer._domainkey.e k=rsa; p=xxxor CNAME to point to mail server and mail server's TXT record will have the final DKIM value. Refer to dns:dkim- (no term)
- Ensure email has header DKIM-Signature that uses
d=e.mydomain.comto match the From: e.mydomain.com
38.38.4. Exchange anti-spam message headers
- https://docs.microsoft.com/en-us/office365/securitycompliance/anti-spam-message-headers
- https://mha.azurewebsites.net/pages/mha.html
- Refer to ms:scc
- Exchange Online Protection (EOP) inserts to each email
CIP:[IP Address]- The connection IP
IPV:NLI- IP was not listed on any IP reputation list
SFV:SFE- Filtering was skipped and the message was let through because it was sent from an address on an individual's safe sender list
SFV:SPM- Indicates that the message was marked as spam because of the EOP spam filters
SFV:BLK- Indicates that the message was marked as spam because the sending address is on the recipient's Blocked Senders List
SFV:SKS- Indicates that the message was marked as spam prior to the content filter. This could include a mail flow rule (also known as a transport rule) marking the message as spam. Run a message trace to see if a mail flow rule triggered which may have set a high spam confidence level (SCL)
SFV:SKB- Indicates that the message was marked as spam because it matched a block list in the spam filter policy
SFV:BULK- Indicates that the Bulk Complaint Level (BCL) value located in the x-microsoft-antispam header is above the Bulk threshold that has been set for the content filter. Bulk email is email which users may have signed up for, but may still be undesirable. In the message header find the BCL (Bulk Confidence Level) property in the X-Microsoft-Antispam header. If the BCL value is less than the threshold set in the Spam Filter, you may want to adjust the threshold to instead mark these types of bulk messages as spam. Different users have different tolerances and preferences for how bulk email is handled. You can create different policies or rules for different user preferences
SCL:[value]- Spam Confidence Level
- -1
- non-spam coming from a safe sender, safe recipient or safe listed IP address (trusted partner). Deliver to inbox folder
- 0, 1
- non-spam because the message was scanned and determined to be clean. Deliver to inbox folder
- 5, 6
- Spam. Deliver to Junk Email folder
- 7, 8, 9
- High confidence spam. Deliver to Junk Email folder
- provides addtional info about bulk mail and phishing
BCL- Bulk Complaint Level
- 0
- message isn't from a bulk sender
- 1, 2, 3
- message is from a bulk sender that generates few complaints
- 4, 5, 6, 7
- message is from a bulk sender that generates a mixed number of complaints
- 8, 9
- message is from a bulk sender that generates a high number of complaints
PCL- Phishing Confidence Level
- 0-3
- message content isn't likely to be phishing
- 4-8
- message content is likely to be phishing
- -9990
- message content is likely to be phishing. Exchange Online Protection only
- EOP inserts email authentication results
You can use
- Exchange Admin Centre > Protection > Spam filter to create a block or allow list based on sender email address or sender domain
- Exchange Admin Centre > Protection > Connection filter to create a block or allow list based on IP
- only use it when filter criteria is complex e.g checking message headers or the names of attachments or add complex actions e.g. adding a disclaimer to the message or applying a time period where the rule is active
38.38.5. Best Practice, Spam Score
Other Spam Score Tools
- https://sendgrid.com/blog/5-ways-check-sending-reputation/
- SenderScore.org
- TalosIntelligence.com
- ReputationAuthority
- BarracudaCentral
38.38.6. Preview text
Preview text is the hidden text that is the first element in the body and it should be at least 90 characters long. Img alt attribute values.
<div style="display:none;font-size:1px;color:#333333;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;"> Insert preview text here. </div> <!-- Insert ‌ hack after hidden preview text. 10 characters --> <div style="display: none; max-height: 0px; overflow: hidden;"> ‌ ‌ ‌ ‌ ‌ </div>
38.38.7. Klaviyo
- Email marketing platform used by big companies
- kmail-lists.com
38.38.8. Send test HTML email for free
https://htmlmail.pro/ use your Gmail to send HTML emails.
38.39. Payment Gateway
38.39.1. Chase E-xact Gateway
Chase Paymentech
https://support.e-xact.com/hc/en-us/articles/360000632374-Hosted-Checkout-Integration-Manual
Chase Payment Gateway
- Use a custom form and post to the following destination to bring up the Chase Payment Form
- Test mode form goes to https://rpm.demo.e-xact.com/payment
- Normal mode form goes to https://checkout.e-xact.com/payment
Payment Page Settings General Return to Your Site URL is hard coded. It's shown when a user comes to Payment Page and sees a Return to Website if the user doesn't want to proceed. And it also serves as a go back link when timeout happens. Receipt Page AUTO-POST, AUTO-GET don't display a receipt on Chase Payment Gateway but instead post the transcaction result to your website POST, GET display a receipt on chase Payment Gateway and a button including a link to GET or POST back to your website. Unfortunately, this link cannot be dynamic.
Relay Response Transaction result is POST to your site and your site responds with HTML which E-xact later displays. The response is usually a custom receipt. If response is not received in 25 seconds, E-xact will display its own receipt. You can give a dynamic relay response url in the submit form rather than a static url setup in E-xact Payment Page setting page. The form field is x_relay_url If replay response is required, in submit form you should add a hidden form field <input name="x_relay_response" value="TRUE" type="hidden"> Silent Post Same as replay response but the response is ignored. The sequence of silent post and relay response is not guaranteed!
Chase Payment Gateway Form Create a Payment Page and you will get transaction_key, page_id, response_key
Do not expose transaction_key, page_id, response_key!!
Form field for submit (required fields) x_login :: page_id x_invoice_num :: your internal order id $order_number x_fp_sequence :: Random number e.g. your internal order id. $order_number x_fp_timestamp :: Requests expire after 15 minutes. Time in seconds. php :: time(). $x_fp_timestamp x_amount :: positive number. $x_amount x_fp_hash :: string. HMAC-MD5 hash from the transaction_key and concatenation of the values for x_login, x_fp_sequence, x_fp_timestamp, x_amount and (if given) x_currency_code. Separated by ^. php : hash_hmac('MD5', $page_id. '^' . $order_number. '^' . $x_fp_timestamp. '^' . $x_amount. '^CAD' , $transaction_key); x_show_form :: 'PAYMENT_FORM', constant
Post back data and fields https://hostedcheckout.zendesk.com/hc/en-us/articles/114094066114#10.1 x_response_code 1 for approved, 2 for process but not approved, 3 for not processed including cancel
x_response_reason_code 1 for approved, 2 for declined, 3 for error x_response_reason_text
x_invoice_num :: the same as x_fp_sequence in TandT case x_fp_sequence
x_test_request boolean. If it's in test mode, true exact_ctr string. Receipt string. Successful or failed, always return a string. Always show! exact_issname string. Issuing Bank name exact_issconf string. Issuing Bank Confirmation Number x_fp_timestamp x_MD5_Hash
You need to validate the post back value if $_POST['x_MD5_Hash'] == md5($response_key. $_POST['x_login'] . $_POST['x_trans_id'] . $_POST['x_amount']);
38.39.2. Moneris Payment Gateway
https://developer.moneris.com/More/Testing/Testing%20a%20Solution https://developer.moneris.com/More/Testing/Penny%20Value%20Simulator
- Visa
- 4242424242424242 4502285070000007
- MasterCard
- 5454545454545454
38.39.3. Stripe
- https://woocommerce.com/products/stripe/
- as long as your yearly charges are below $1 million
- Credit and debit cards
- pay 2.9% + 30 cents per successful transaction
- (no term)
- https://stripe.com/en-ca/pricing
- No China
- Refund, recurring payments
- Credit cards, debit cards, Apple Pay, Google Pay, Alipay and Payment Request API support
- Team
- An admin (one email address) can create multiple Stripe accounts
- Admin can add members to manage a Stripe account
- Products
Types
- Goods
- use Orders API
- Services
- subscriptions. Has following parameters
- name
- type
- metadata
- Plans
- Has following parameters
- product: e.g. ID of an existing product to associate with the plan
- amount to charge a customer per subscription per interval
- day, week, month and year. Use
interval_countto set e.g. 30 days - unique ID, auto generated by Stripe
- can't change amount, currency and interval after a plan is created. Can change metadata and nickname
- Has following parameters
38.39.4. PayPal Powered by Braintree on WooCommerce
- PayPal Powered by Braintree
- https://woocommerce.com/products/woocommerce-gateway-paypal-powered-by-braintree/
- 2.9%+30 cents per successful credit card or digital wallet transaction (e.g. Apple Pay)
- Credit cards and debit cards
- Payment from buyers' PayPal accounts
38.39.5. Zum Rails
- https://zumrails.com/
- API Doc
38.40. Market Research market research:firm
- Refer to market research:service
- Forrester
- Gartner
- https://vividata.ca/
- https://hashtagpaid.com/">AI to find influencers
- tech stack:g2
- Direct Marketing (US and Canada)
- https://www.cardinalpath.com/
38.41. W3C
- Working Draft
- Last Call
- Candidate Recommenation (CR, Standard)
- Proposed Recommendation (PR)
- Recommendation (REC)
- Working Group Note, Interest Group Note (NOTE) which may make Editor's Drafts (ED, API's may take this as a strong signal..)
38.42. Adobe
38.42.1. Acrobat Pro DC
38.42.1.1. Open the same file in separate windows
Window -> New Window
38.42.2. Creative Cloud
If you have error 146 and an app update failed, you might need to reinstall Creative Cloud Desktop App and then update individual app.
38.42.3. Photoshop
38.42.3.1. Image Resize
- Preserve Details (enlargement)
- Bicubic Smoother (enlargement)
- Bicubic Sharper (reduction)
Resize an animated gif Window > Workspace > Motion
Crop if you like.
File > Export > Save for Web (Legacy) > Resize it there
38.42.3.2. Make a lighter or darker color
- Shade
- mix with black
RGB: Multiply each component by 1/4, 1/2, 3/4 etc. of its previous value. The smaller the factor, the darker the shade
$newR = $currentR * (1 - $shade_factor); $newG = $currentG * (1 - $shade_factor); $newB = $currentB * (1 - $shade_factor);
- lower the Value / Brightness or increase the Saturation
- lower the Lightness
- Tint
- mix with white
RGB: Calculate (255 - previous value) for each component, multiply that by 1/4, 1/2, 3/4, etc. and add that to the previous value. The greater the factor, the lighter the tint
$newR = $currentR + (255 - $currentR) * $tint_factor; $newG = $currentG + (255 - $currentG) * $tint_factor; $newB = $currentB + (255 - $currentB) * $tint_factor;
- increase Value / Brightness or lower the Saturation
- increase the Lightness
- Tone
mix with with gray
$newR = $currentR + ($aR - $currentR) * $alpha; $newG = $currentG + ($aG - $currentG) * $alpha; $newB = $currentB + ($aB - $currentB) * $alpha; // Where // black = (aR, aG, aB) = (0, 0, 0) // white = (aR, aG, aB) = (255, 255, 255) // greay = (128, 128, 128)
- (no term)
- PHP Gist
- (no term)
- To easier and more granularly manipulate shade and tint, convert Hex/RGB to HSL, adjust the lightness and then convert back to Hex/RGB
38.42.4. Illustrator
38.42.4.1. Setting
Preferences > General > enable Scale Strokes & Effects
Control Panel > Document Setup When nothing is selected, on the top below menu, you will see a Control Panel. Document Setup can set Units and Bleed.
View > Smart Guides Show Smart Guides (alignment suggestion)
View > Show Grid Show grids inside and outside artboards
38.42.4.2. Panels, Workspace, Artboard
Add extra panels Window Menu
- Align along with Transform and Pathfinder
- Info without Navigator
- Type > Character panel
Default Panels
- Layers Panel
- Change path color
- Template and Dim images to
- That layer has raster images. Used as a background to draw vector path on top of it
- Move objects inside or to a different layer
- Artboards Panel, Artboard Tool
Shift + O
- Change canvas size, orientation and other settings which you see when creating a document
- May also change the current artboard in the Artboard Tool
- Export individual artboard to different file format File > Export > Export for Screens
- Show Center Mark (center point)
Save as a new workspace. Reset to default workspace.
38.42.4.3. Hand, Zoom, Screen mode, View, Rulers, Guides
Hand Tool H or Spacebar to temporary change to hand tool
Zoom tool (z)
Alt + Clickzoom outCtrl + 0view whole or double click on hand toolCtrl + 1view at 100% or double click on zoom tool
Cycle through screen modes for presentation F
A View is a view which shows a portion with correct zoom and boundaries.
View Menu > Create a new view
Show Rulers Ctrl + R
- Drag from one ruler to create a guide line
- Or double click on the ruler to create a guide
- Lock Guides in View Menu > Guides > Lock Guides
- View > Guides > Make Guides
38.42.4.4. Repeat the previous action
Ctrl + D e.g.
- Duplicate an object after Alt + Drag
- after scaling with the Copy action.
38.42.4.5. Move, Select
Shift + Arrow :: move 10px
Ctrol + click on an object :: temporary select an object Ctrol + Drag :: temporary select multiple objects
38.42.4.6. Pencil and Brush tools
Free hand draw. Pencil Tool (N) can only apply regular stroke. Double click on the Pencil Tool on the left to changeL
- smoothness
- Closing path pixel amount
When about to close the path, hold down Alt. When you see a circle in the cursor, release the mouse. The path will be closed.
Redraw part of the path to reshape part of the path
Brush (B) is the same as pencil but can add pattern and image in path.
- Window > Brush
38.42.4.7. Basic Shapes
- Line, arc, spiral, rectangular grids, polar grid
In the left menu
- line (\)
- arc
- spiral (Shift, Up and Down to increase # of rings )
- rectangular grids
- polar grid
- Rectangle, Ellipse
In the left menu
- Rectangle tool (M)
- Rounded Rectangle Tool
- Up or Down to change the number of sides
- Up or Down to change the number of angles.
- Flare Tool
For any shape tool
Alt + Clickto draw a new shape with the same center point of another shapeCtrl + Dragto make the shape fatter or skinnier.
38.42.4.8. Drawing Modes
Shift + D :: change drawing mode: normal, behind, inside
38.42.4.9. Transform Object
- Change Anchor Points or Control Handles
Direct Selection Tool
AAlt + Dragon anchor point to change only on one side
- Group, Ungroup Objects, Isolation mode, Lasso Tool
Ctrol + GGroup multiple objectsShift + Ctrol + GUngroup
Isolation mode: Double click on a group to go to sub group
Lasso Tool
Q- Manual select multiple objects and then group them.
- Duplicate Object
Alt + Drag :: with
Shiftto duplicate it horizontally or vertically Repeat the previous actionCtrl + D - Offset an object
Offset anchor points to create another object Object > Offset Path
- Scale, Shear
Scale Tool
S. Double click to scale numerically. May also change the reference point to scale to withtout Double click.Shear Tool. Double click to shear numerically.
- Rotate and Reflect Object
Rotation Tool
RChoose another reference point to rotateAlt + Dragwhile rotating and release will make a copy after rotation Then may repeat the action to make several copiesCtrl + DReflect Tool
OSymmetry When the tool is selected, the reference point by default is the center point. Change the reference point and then drag. UseShiftto rotate vertically or horizontally Hold downAltwhile drag and then release to copy the reflection objectMay also right click on the object, select Transform > Reflect and select copy. Later move that object in place
- Free Transform Tool
Mouse operations only
ERotate, scale, change aspect ratio - Transform Each
Object Menu > Transform > Transform Each
Ctrl + Alt + Shift + DTransform based on one of the 9 points.
38.42.4.10. Fill and Stroke
Default Fill and Stroke D
No fill \
Select multiple objects, press I, then select a color. All objects will be filled in that color.
- Gradient
Select an object, go to Window > Gradient, select a Type.
On the left menu, select Gradient Tool
G, then draw line inside the object.May apply gradients to both stroke and fill for an object.
- Stroke
Show outline only
Ctrl+YClick on Stroke on the top menu (Control Panel), when a path/object is selected.May change the width of a stroke partially or dynamically using Width Tool
Shift + W
38.42.4.11. Color, Swatch
- Color Setting
Edit > Color Settings North America General Purpose 2 RGB: sRGB IEC61966-2.1 CMYK: US Web Coated SWOP v2 Choose Preserve for RGB and CMYK Enable all Ask When Opening, Ask When Pasting
- Process, Global, Spot Color
Process color means that color only applies to an object. Changing that process color only changes color of that object. Changing a global color means all objects using that global color will change color.
To create a global color Open Window > Swatches Apply a color for an object. Select or create a group. New Swatch, select Global
Spot color is usually provided by a vendor/company which, for example, provides specific ink. To further ensure the spot color is always accurate on finished products (e.g. prints).
- Import Export Swatch
.ase file. Window > Swatches
38.42.4.12. Appearance
Window > Appearance Add overall effects on all objects or on a specific object Change attributes: Stroke, Fill, Opacity May stack stroke and fill effects. May use pattern in Fill Add Live Effects (fx) or Effect (top menu): Drop Shadow, Warp etc. Save appearances as graphic styles Select an object with some Appearances, then open Window > Graphic Styles, create new Apply that new Graphic Style on another object in the same .ai file. In the Graphic Styles window, Select unused graphic styles, and remove. Save the rest Graphic Styles, and hit Save Graphic Styles as Library. Open a new ai file, in the Graphic Styles Window, load User Defined. Then you can apply the Graphic Style across ai files.
38.42.4.13. Complex Shapes, Compound Path, Pathfinder, Shape Builder
Compound Path
- 2 Paths: Inner and outer circles
- Select both paths and Ctrl+8 to make a compound path
- Later double click on the compound path, inner circle can be resized!
Pathfinder
- Window > Pathfinder. You can't change the paths after it's applied
- Shape modes: the result is a single path of the end shape
- Unite
- Back path mius front path. Punch a front path hole in back path.
- Intersect
- Front + Back - intersect
- Pathfinders mode: the result is a bunch of paths that form the end shape
- Divide
- Front + Back but with shapes cut at any intersection
- (no term)
- Trim
- (no term)
- Merge
- (no term)
- Crop
- (no term)
- Outline
- (no term)
- Minus Back
Shape Builder
- Left menu: Shift + M
- Result is like Pathfinder's Pathfinders mode.
- Select both paths and draw straight line to indicate which area should bring front
- Draw straight line while holding
Altis to subtract
38.42.4.14. Eraser
Eraser fixes path around edges. Shift + E [ or ] to increase the size
Make sure to select a path then erase. Otherwise all paths will be erased.
38.42.4.15. Pen Tool
Pen Tool P
- Point, click and move to create straight line
Alt + Dragto bend any path at any point- Draw curve line with Pen Tool
- Start point: Click and drag to the direction that the curve starts
- End point: click and drag to the opposite direction that the curve ends
- Change handle pdirection of an anchor point
- Direct Selection Tool
Aand change the handle direction
- Direct Selection Tool
- Remove "whip": remove the handle of an anchor point
Alt + Clickon the anchor point
- No fill or fill none
\when necessary - Direct Select Tool
Ato change anchor points and handles
Add Anchor Point Tool +
Delete Anchor Point Tool -
Anchor Point Tool Shift + C
- Turn a point into anchor point (no handle, angle)
Click + Dragto create 2 handles
38.42.4.16. Type Tool
T
Convert between Point Type and Area Type Type > Convert to Point Type
- Point Type
- one line. It scales when it's resized
- Area Type
- Fit text into a box. It doesn't scale when it's resized
Panels
- Window > Character
- Character, Paragraph, OpenType
- Window > Character Styles
- Used to save Character or Paragraph styles defined above
- Character Styles, Paragraph Styles
Wrap text around an object
- Select the object that text should wrap around
- Object > Text Wrap > Make
Object > Text Wrap > Text Wrap Optionsto change the offset- Right click on the same object, Arrange > Bring to Front
Type on a Path Tool
- e.g. curve path
Change Font Size Shortcuts
shift + ctrl + > or <2 points at a timeshift + alt + ctrl + > or <10 points at a time
Change Tracking of Font (horizontal space between characters)
opt + right or left arrow
Paragraph spacing opt + up or down arrow
Adobe Typekit :: Sync Fonts with local Desktop as long as Creative Clouds are installed locally
Turn Type into Paths Type > Create Outlines
38.42.4.17. Copy from InDesign
- Select objects in InDesign and paste in AI.
- View > Outline, you will clipping mask. Get rid of it!
- Object > Clipping Mask > Release
- Select the outest outline and delete it.
- View > Preview to turn off Outline view.
38.42.4.18. Resize canvas to selection
Object > Artboards > Fit to Selected Art
38.42.4.19. Raster Image
Place an image File > Place
The image is linked from Illustrator to the image file.
Illustrator can see any changes on the image file.
Links Panel Window > Links
Embed and Unembed (relink)
Clipping Mask
- Draw a circle shape on top of the image
- Select both the circle and the image
Object > Clipping Mask > Makeor right clickMake Clipping Mask- Double click to change the circle or image
- Later right click on the image,
Release clipping maskto remove the circle mask
Image Trace
Select the image, Window > Image Trace
Convert Image Tracing to Paths
Select the Image Tracing object and Object > Expand or Expand in the top Image Tracing control panel
Once expanded, image cannot be traced again.
38.42.4.20. Crop image
You CAN crop raster images in Illustrator.
Draw a rectangle above the raster image. Select both the rectangle and the image choose Object > Clipping Mask > Make. change blending mode to "darken" from transparency tab choose Object > Flatten Transparency..> OK Choose Object > Expand… Now you have trimmed raster object.
38.42.4.21. Export Assets
Window > Asset Export. Add objects one by one to Asset Export
38.42.4.22. Export SVG
Select the object on the artboard then File > Save As > svg
Turn font into path :: Select all text > Type > Create Outlines
CS6
Save Asall objects on one or multiple artboards.- Can't
Exportselection or Export multiple artboards - The artboard size is the viewport size in SVG
CC version
- All or selected artboards can be Exported as SVG.
- Selected objects can be exported separately using
File > Export Selection - Add an object as Asset and Export Asset
- Copy the object and paste in Dreamweaver!
SVG Profile :: SVG 1.1 Fonts.
- Type: SVG. If there's small text, use Convert to Outline which converts all Type to paths
- Subsetting: None
- Embed. Usually don't include raster image in SVG
- Don't check
Preserve Illustrator Editing Capabilitiesbecause it increases file size
Styling :: Inline Style or Presentation Attributes Font :: Convert to Outlines Images :: Preserve Minify but not Responsive
38.42.4.23. Scripts
Save script file in C:\Program Files (x86)\Adobe\Adobe Illustrator CS6\Presets\en_US\Scripts\
Restart AI and the script can be executed under File > Scripts >
38.42.5. Color CC
https://color.adobe.com/create/color-wheel/ https://www.lynda.com/Creative-Cloud-tutorials/Kuler-Essential-Training/136175-2.html
Analogous Create variants of the Base color which is always the center point. All 5 colors have the same saturation but differ in hue.
Monochormatic Same Hue value, but different saturation and brightness values.
Triad Create contrast colors.
Complementary Opposite colors.
Compound 2 sets of opposite colors and for each set create another analogous color
Shades Only different in brightness.
Use Mobile App: Adobe Capture to capture colors.
38.43. Digital Edition
38.43.1. Issuu
38.43.2. Pugpig
- Clients
- GQ, Scientific America, The Economist, The Sun
38.44. Publication
38.44.1. Production
38.44.1.1. ppi Media
- Clients
- New York Times
38.44.1.2. AdFlow Systems GmbH
- German
- Clients
- Montreal newspapers
38.44.2. Market research market research:service
- Refer to market research:firm
- Cision
Earned media management software and services
- Distribute content: print, radio, television, web and social media
CEDROM-Sni
- Digital reproduciton rights (buy and sell copyright)
- Digital media monitoring and analytics
- Target key influencer
- Measure impacts
- https://www.comscore.com/
- 30M Canadian Digital Audience, 11M Tablet, 18M Smartphone, 28M Desktop - 2017
- Computer usage peaks at 7-8AM and all devices usage peak at 5-8PM - 2017
- Mobile app usage is 86% compared to Mobile Web is 16% - 2017
- Total time spent for all internet across devices: Desktop:38%, Smartphone App:38%
- Total time spent for social media acrss devices: Desktop:21%, Smartphone App:50%
- Quantcast
- AI driven, audience insight from all websites use Quantcast. e.g.
- What sites a reader visited
- Keywords searched
- Purchase habits
- The database is called Quantcast Intelligence Cloud (QIC) powered by Q engine
- AI driven, audience insight from all websites use Quantcast. e.g.
38.45. CRM
38.45.1. HubSpot
38.45.1.1. Understanding the CRM
- An object has properties
- Objects
Deal
- associate with
- a contact or company
- line items
- associate with product
- Engagement
- Ticket
- Timeline Event
- Deals API
- https://developers.hubspot.com/docs/api/crm/deals
- https://legacydocs.hubspot.com/docs/methods/deals/deals_overview
Ticket
- associate with
- a contact or a company
- Engagement
- Deal
- Timeline Event
Contact
- associate with
- company
- Deal
- Engagement
- Ticket
- Analytics Event
- Timeline Event
Company
- associate with
- Contact
- Deal
- Engagement
- Ticket
- Timeline Event
Engagement
- associate with
- Ticket
- a contact or company
Line item
- associate with
- Deal
- Product
Custom objects
- Requires any Enterprise subscription
- Go to any native object, and click the down button to see custom objects
- https://developers.hubspot.com/docs/api/crm/crm-custom-objects
- Requires any Enterprise subscription
38.45.1.2. Pricing
- Categories
- Marketing Hub
- Sales Hub
- Service Hub (Custom Service)
- CMS Hub
- Operations Hub
- Level
- Free Tools
- Starter
- Professional
- Enterprise
38.45.1.3. Workflow (Webhooks)
- Intro
- https://developers.hubspot.com/blog/implementing-webhooks-in-hubspot
- (no term)
- Part of Operations Hub
- Free is already powerful
- Webhooks API
- https://developers.hubspot.com/docs/api/webhooks
- (no term)
- Use UI to setup a webhook to trigger events
- Workflows on the top or https://app.hubspot.com/workflows/%7Baccountid%7D/view/all
38.45.1.4. API
- Engagement Entity
- https://legacydocs.hubspot.com/docs/methods/engagements/engagements-overview
- New https://developers.hubspot.com/docs/api/crm/engagements
- Notes, Emails, Calls, Tasks, Meetings
- Attachments are part of a note
- To get engagements of a deal
- {{gateway}}/crm-associations/v1/associations/{{deallD}}/HUBSPOT_DEFINED/11
- To get one engagement
- Files
38.45.1.5. Different Accounts
- https://developers.hubspot.com/docs/api/account-types
- Standard
- App developer accounts
- Developer test accounts
- Up to 10 developer test accounts can be created under a app developer account
- It's an Enterprise level account! Expires every 90 days
- To renew a developer test account, switch to the app developer account, click on Testing tab in between Apps and App Marketplace and select a developer test account to renew trial
38.46. Automations
38.46.1. Zapier
- Build a Zapier integration https://zapier.com/platform
- https://www.youtube.com/watch?v=CQ4j2GY-nbc&t=612s&ab_channel=Zapier
- https://developer.zapier.com/cli-guide/testing-your-app
- https://www.flowfinity.com/kb/zapier-write-data.html
- https://platform.zapier.com/partners/lifecycle-planning
- https://platform.zapier.com/docs/zapier-intro
38.46.1.1. CLI
- Dockerfile:
npm install -g zapier-platform-cli
FROM node:16 # Place global NPM dependencies in a non-root user directory ENV NPM_CONFIG_PREFIX=/home/node/.npm-global # optionally if you want to run npm global bin without specifying path ENV PATH=$PATH:/home/node/.npm-global/bin # Install global NPM modules RUN npm install --global zapier-platform-cli COPY .zapierrc /root
- Example App
git clone https://github.com/zapier/zapier-platform-example-app-github.git cd zapier-platform-example-app-github npm install - Environment Variable
project/.envfile
- Refer to a variable
process.env.MY_ENV_VAR_1
- GitHub
zapier-platform
- CLI Commands
zapier login --sso
- Create a Deploy Key
- https://zapier.com/developer/partner-settings/deploy-keys/
- The URL will be given when you first run
zapier login --sso - The credential is saved to
/root/.zapierrcgiven that the current user isroot
# Use single sign-on zapier login --sso
- Create a Deploy Key
zapier register
- https://platform.zapier.com/reference/cli#register
- Create a new integration app and link the local app (link only once) to that integration
.zapierapprcis created and it looks like this{ "id": 123456, "key": "App123456" }- If
.zapierapprcexists, it will ask if you want to update the currently-linked integration as opposed to creating a new one - After
zapier register,zapier pushcan be run to build and upload the new integration for use in the Zapier editor.
zapier push
- Before push, better do
zapier validate - After push, the follow files are created/updated under
./build- build.zip
- source.zip
- Before push, better do
zapier integrations
- If you find the app and the version is not listed, you will need to re-login again
zapier login --sso
- If you find the app and the version is not listed, you will need to re-login again
zapier logs
# Recent HTTP traffic to my API zapier logs --type=http # More detailed zapier logs --type=http --detailed # In the app code, throw custom debug messages using `z.console.log("my debug message")` and catch it in the logs: zapier logs --type=console # More opptions for the zapier logs command zapier help logs
- Field types
38.46.1.2. Rate Limits and Throttling
38.46.1.3. Building Zaps
38.47. Form Service
- JotForm
- Video uploading
- 38.39
- Dropbox
- Voting
38.48. Survey Service
- SurveyMonkey
- Survey Monkey
Pricing
- Business Plan
- Team Advantage $27 CAD/user/month
- Team Premier $75 CAD/user/month
- Seat Subscription Renewal (1 user/year, $780 from 2019 to 2020. Manual payment)
- Personal Plan
- Phone Support
- If the plan has phone support, fill up this contact form with
I need more help, then support will call you
- If the plan has phone support, fill up this contact form with
- Question Types
- multiple choices, short answer, dropdowns and checkboxes
- Sliders, payment-acceptance fields (via Stripe) and dropdown matrices
- File upload
- (no term)
- SurveyGizmo
- Plan: https://www.surveygizmo.com/plans-pricing/
- Full Access - $199/month
Reports
- TURF report: Total Unduplicated Reach and Frequency
- 100 respondents, 90 like cake A,B,C, the other 10 like cake X and Y. Restaurant owner can choose 3 new cakes to make. Result: A (or B, C), X, Y
- Integration
- API access
- Salesforce Marketing Cloud
- Google Analytics
- Google Spreadsheets
- Webhooks
Sharing
- Permalink
- iframe,
- Buy US respondents!
- Download, install on mobile and desktop, collect responses offline
- QR code
- Website Intercept
- Banner (top/bottom), modal dialog, inline intercept
- JS code on every page
sg_beacon( 'init', 'xxxyyyzzz')inits and starts sending beacon- During init, check trigger. Trigger requires another init call or page refresh/new pageview
- Plan: https://www.surveygizmo.com/plans-pricing/
- (no term)
- wp:plugin:crowdsignal
38.49. Merchandise
38.49.1. Amazon Associates
- Join
- Have to have an Amazon account
- Payee info
- Top level domains of websites and/or mobile apps on which you plan to display banners, widgets, Special Links, or other ads from Amazon Associates
38.50. HR
- Use Radar chart or spider chart to compare candidates
- Criteria
- Education
- Knowledge, Skills and Abilities (KSA)
- communication
- Behavior
- Creativity
- Self-directed
- Relationship builder
- Tolerance of ambiguity
- Flexibility
- Persuasiveness
- Attention to detail
- Results orientation
- Organizational Compatibility
- Career path relevancy
- Genuine interest
- Values/mission alignment
- Commitment to higher education
- Civically engaged
- Diversity
- Multilingual
- Nontraditional career path
- International experience
- Equal opportunity initiatives
- Experience working with diverse communities
- Experience working with people who require reasonable accommodation
- Previous experience managing conflict across differences
38.51. Training
38.51.1. Microsoft Virtual Academy MVA
Free https://mva.microsoft.com Put in the Exam code, e.g. 70-480 to get free training resources
38.51.2. Microsoft Certification exam list
38.51.3. Microsoft Certification List and Study Group
38.51.4. Microsoft Online Proctored Exams
https://www.microsoft.com/en-us/learning/online-proctored-exams.aspx Install the software and do a hardware test. Sign in http://microsoft.com/learning to take an exam half an hour early Click "Your benefits & exams" then "Start a previously scheduled online proctored exam"
38.51.5. HackerRank
38.51.6. CoderByte
38.52. Work Ethics
38.52.1. Remove Credentials
- Windows Credential Store Panel
- Chrome
- Git remove remotes
git remote rm lithis also deletes all branches of that remote ~/.ssh/configrm -rf /mnt/e/li/- phpStorm remove all projects
38.53. TeamViewer
1)完整卸载并删除现有TeamViewer安装过的版本 2)删除:%AppData%\Teamviewer、%tmp%\TeamViewer 3)删除:C:\Users\Administrator\AppData\Local\TeamViewer 4)删除:HKCU\Software\TeamViewer、HKLM\SOFTWARE\TeamViewer 5)下载Teamviewer对应版本,安装方式选个人非商业用途,然后打补丁即可!
如果想让破解补丁用于便携版,可装一次安装版,然后打完补丁后把下面三个文件: TeamViewer.exe、TeamViewer_Service.exe、TeamViewer_Desktop.exe 复制出来替换到便携版即可!
38.54. Font
- WOFF
- font:woff
- supported in Edge and modern browsers. For web, always include WOFF (Web Open Font Format) > TTF > OTF
- It's good for SEO
- It has TrueType Hinting and OpenType Features
- EOT
- IE 11 and less
- OTF or TTF
- For local computers, install OTF (OpenType Font) over TTF (TrueType Font)
OTF is newer and based on TTF but it's much smaller
- OpenType Features
- TTF rendered in different devices has non-equal pixels
- TrueType Hinting
- Adobe Fonts and Adobe Edge Web Fonts provides it for better rastering on pixel-based digital device
- (no term)
- Refer to css:font-face
38.54.1. Abbreviations Foundary
- https://www.extensis.com/blog/fonts/abbreviations-font-names/
- usually in the form of one or two letters at the beginning or end of the name (LT, MT, A, BT, FB, URW). “Foundries” are the companies that create fonts, a term going back to the days of metal type
- comes at the end of a name (Cyr, Grk, CE). Generally this only applies to older fonts where a separate font was issued for different languages. In most cases, newer fonts put all the languages in a single font
- (Text, Display, Poster/Caption, Small Text, Regular, Subhead, Display)
38.54.2. TS: Not a valid font file Windows
Turn off firewall or try font converter to convert the font and then install
38.54.3. Line height
Fonts rendered different line height across browsers is because Line Height Adjustment option is not in the font.
Line Height Adjustments
- Best (Use the best method to normalize the line height for this font)
- 120% (Redefine the line height as 120% of the point size)
- Automatic (Distributes OS/2.Typo values across ascender, descender and line gap)
- *Bounding Box (Match the bounding box of the glyphs, line gap will always be 0)
- Native (Use the line height as defined in the font, results may differ between browsers)
Use Bounding Box.
https://iamvdo.me/en/blog/css-font-metrics-line-height-and-vertical-align
p { * font metrics * –font: Catamaran; –fm-capitalHeight: 0.68; –fm-descender: 0.54; –fm-ascender: 1.1; –fm-linegap: 0;
* desired font-size for capital height * –capital-height: 100;
* apply font-family * font-family: var(–font);
* compute font-size to get capital height equal desired font-size * –computedFontSize: (var(–capital-height) / var(–fm-capitalHeight)); font-size: calc(var(–computedFontSize) * 1px);
* compute line-height:normal and content-area's height * –lineheightNormal: (var(–fm-ascender) + var(–fm-descender) + var(–fm-linegap)); –contentArea: (var(–lineheightNormal) * var(–computedFontSize));
–distanceBottom: (var(–fm-descender)); –distanceTop: (var(–fm-ascender) - var(–fm-capitalHeight));
–valign: ((var(–distanceBottom) - var(–distanceTop)) * var(–computedFontSize));
–line-height: 3; line-height: calc(((var(–line-height) * var(–capital-height)) - var(–valign)) * 1px); }
38.54.4. Open source alternative, identify font
38.54.5. Variable Fonts
38.54.6. Windows Font List
38.54.7. iOS Font List
38.54.8. Extensis
https://www.extensis.com/support Download Universal Type Client (6.1.5) not Universal Type Server 6 Core Client
38.54.9. Adobe Fonts former Adobe TypeKit (AF)
- https://fonts.adobe.com/about
- Pricing
- Complete library of fonts is included with
- Creative Cloud All Apps subscription
- Creative Cloud any single-application subscription
- Each seat of a Creative Cloud team subscription
- the Photography Plan
- Some or full font library is included in some educational licensing programs
- Complete library of fonts is included with
- Font Pack
- Showcase a group of fonts, then you can select individual font to fav, add, activate
- Activate a font to use on desktop apps with a subscription plan
- Web project
- No limits on pageviews and domains for any web projects
- Character set or subset
- Default
- English, French, German, Italian, Portuguese and Spanish
- (no term)
- All Characters
- Dynamic Subsetting
- required for East Asian web font
38.54.10. Adobe Edge Web Fonts (AEWF)
- https://edgewebfonts.adobe.com/
- Free!
- Source Sans Pro and Source Code Pro
38.54.11. Google Font (GF)
- Can be downloaded and installed on Desktop as .ttf and for website .woff or .woff2
https://fonts.googleapis.com/css?family=Tangerine|Inconsolata|Droid+Sanshttps://fonts.googleapis.com/css?family=Tangerine:bold,bolditalic|Inconsolata:italic|Droid+Sanshttp://fonts.googleapis.com/css?family=Source+Sans+Pro:300,400,600,300italic,400italic,600italicSubset / scripts, Latin subset is always included if available and need not be specified
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto+Mono&subset=greek">
https://fonts.googleapis.com/css?family=Inconsolata&text=Hello%20World- effect">
https://fonts.googleapis.com/css?family=Rancho&effect=shadow-multiplefont-effect-class prefix<div class="font-effect-shadow-multiple">This is a font effect!<div>
38.54.12. FontForge
- http://fontforge.github.io/en-US/
- Element > Font Info > General to get Em Size
- Element > Font Info > OS/2 to get
- Win Ascent, Win Descent for Windows
- H Head Ascent, H Head Descent for Mac OS
- Capital Height
- X Height
- Typo Line Gap and H Head Line Gap
38.54.13. Social Media Icons wp:plugin:simple-social-icons
38.54.14. Monospace
i1lI|!j oO0 e3 4AF f2 7TJyt __ ;: "'` Xx* [{( sS xX cC jJ uU vV zZ
abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789
~`!@#$%^&*()-_"'
[{()]}\/><,.?
38.54.14.1. Monoid
38.54.14.2. Hasklig
https://github.com/i-tu/Hasklig extends Source Code Pro
38.54.14.3. Source Code Pro
- https://github.com/adobe-fonts/source-code-pro/releases
- Install TTF on Windows
38.54.15. Material Font, GF
<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"> <!-- IE 10+ --> <i class="material-icons">face</i> <!-- For browsers do not support ligatures, --> <i class="material-icons"></i>
https://fonts.google.com/icons (https://material.io/icons/) and corresponding codepoint index
Recommended sizing and color setting
/* Rules for sizing the icon. */ .material-icons.md-18 { font-size: 18px; } .material-icons.md-24 { font-size: 24px; } .material-icons.md-36 { font-size: 36px; } .material-icons.md-48 { font-size: 48px; } /* Rules for using icons as black on a light background. */ .material-icons.md-dark { color: rgba(0, 0, 0, 0.54); } .material-icons.md-dark.md-inactive { color: rgba(0, 0, 0, 0.26); } /* Rules for using icons as white on a dark background. */ .material-icons.md-light { color: rgba(255, 255, 255, 1); } .material-icons.md-light.md-inactive { color: rgba(255, 255, 255, 0.3); } /* Custom color*/ .material-icons.orange600 { color: #FB8C00; }
<i class="material-icons md-24">face</i> <i class="material-icons md-dark">face</i> <i class="material-icons md-dark md-inactive">face</i> <i class="material-icons md-light">face</i> <i class="material-icons orange600">face</i>
38.54.16. Poppin, GF
38.54.17. Museo, GF
38.54.18. Roboto Slab, GF
38.54.19. IcoMoon
Upload font to IcoMoon, and you can generate SVG sprite!
38.54.20. Fontello
Open source Font, CSS, SVG sprites. Fontelico Font Awesome Entypo Typicons Iconic Modern Pictograms Meteocons MFG Labs Maki Zocial Brandico Elusive Linecons Web Symbols
38.55. Digital Agency
- Agency
- Modern Tribe
- https://tri.be
- PixelCarve.com
- ONroute.ca, conference, condo, recruitment
- AkaNewMedia.com
- IBAO and other insurance websites
- Parachute
- Intact Insurance
- IronPaper.com
- lead gen
- Unleashed-Technologies.com
- publication ECmag.com
- FosterInteractive.com
- UofT Woodsworth College
- A Nerd's World
- Toronto. Full digital design services. https://anerdsworld.com/
- DigiLite.ca
- shops
- MajorTom.com
- before DriveDigital.ca
- WSI
- Digital marketing services. https://www.wsiworld.com/
- https://upperquad.com/
- creative
- (no term)
- https://18f.gsa.gov/
- https://therefore.ca/
- gallery, exchange
- https://echidna.ca/
- universities, tourism
- (no term)
- https://www.optasy.com/
- (no term)
- https://www.myplanet.com/
- (no term)
- https://devshop.support/
- Awards
- AWW
- https://www.awwwards.com/websites/magazine-newspaper-blog/
- (no term)
- https://digitalpublishingawards.ca/
- (no term)
- Web Marketing Association and awards (WMA)
- WebAward
- WebAward.org
- Society for News Design
- https://www.snd.org/
- The Society of Publication Designers
- https://www.spd.org/
- Webby Awards
- https://www.webbyawards.com
- Code Development
- Envato
- ThemeForest: themes
- CodeCanyon: code, plugins
- VideoHive: video stock
- AudioJungle: audio stock, audio tracks
- GraphicRiver: fonts, icons, web elements, vectors, graphics, logos
- PhotoDune: photo stock
- 3DOcean
- Envato
38.55.1. PodBean
38.55.2. Castos.com
38.55.3. Exercism.io
39. Industry Compliance, Standards
39.1. WCAG, AODA, Web Content Accessibility Guidelines
Checking Tools
- SortSite by powermapper.
- Block pages
- View > Options > Blocks e.g. featherlight.gallery.min.css
- Achecker.Achecks.ca">Check one page only. Used by Ontario Government
- First 50 pages is free
- https://www.siteimprove.com/
- Chrome DevTools or plugins
https://www.w3.org/WAI/WCAG20/quickref/
Keyboard accessbile for non link elements and non form elements
<div onclick="doSumat();" onkeypress="doSumat();" tabindex="">
39.1.1. <a>
Multiple read more on a page
<a href="babymayor.html" aria-label="Read more about Seminole's new baby mayor">[Read more...]</a>
39.1.2. iframe
<iframe title="" style="border:0px;">
frameborder="0" is obsolete.
39.1.3. input, label
if <input> has no <label>, then add title attribute
Remove size attribute for input type file, confirm_email
$html = preg_replace('/(<input type="file" [^>]+) size=".*?"/si', '$1', $html);
39.1.4. pdf
- PDF and Office documents need to set Title, Advanced > Language in Document Properties
- PDF and Office document image has to have title attribute.
- Edit > Manage Tools > add Accessibility to the right pane
- Accessibility > Full Check > Click on items need alternate text to find on page and then doubleclick to select and then right click Tag as Figure
- Accessibility > Autotag Document will retag the whole document. The best approach is to add tag to image/figure because autotag might add links wrong based on text context.
- Do these
- Accessbility > Reading Order > Show Order Panel. Right click on the figure and Tag
- http://www.adobe.com/ca/legal/permissions/icons-web-logos.html#adobereader
<a href="http://www.adobe.com/go/getreader" target="_blank"><img src="get-reader-158x39.png" alt="Get Adobe Reader DC"></a>
- PDF/UA-1 compliance. PDF needs to have this XMP metadata to mark it as PDF/UA-1 compliant
- Tools > Print Production > Preflight, choose PDF Standards or Acrobat Pro DC 2015 Profiles as profile, choose the wrench tool icon (Select single fixups), expand to Document info and Metadata and choose Set PDF/UA-1 entry and then Fix
39.1.5. html:meta:viewport
39.1.6. Skip to content link
Right below <body>
<a id="skip-nav" href="#main-content">Skip to content</a> <main id="main-content"></main>
#skip-nav { position: absolute; top: -1000px; left: -1000px; height: 1px; width: 1px; text-align: left; overflow: hidden; } #skip-nav:active, #skip-nav:focus, #skip-nav:hover { width: auto; height: auto; position:static; overflow: visible; }
39.1.7. Color Contrast Ratio
WCAG 2.0 level AA requires a contrast ratio of 4.5:1 for normal text and 3:1 for large text. Level AAA requires a contrast ratio of 7:1 for normal text and 4.5:1 for large text. https://webaim.org/resources/contrastchecker/ http://leaverou.github.io/contrast-ratio/
39.2. SR&ED: Scientific Research and Engineering Development
- Format
- Technical Uncertainties
- the main objective of this project was to …
- Work Done
- we hypothesized that it would be possible to do … in order to achieve the main ojective
- the project was completed? Y/N
- Technological Advancements
- we aimed to achieve the following: …
- Technical Uncertainties
- Besides the format, we need to explain:
- Why is the problem you are solving unique
- What are the known technologies?
- Why is it not simply a matter of engineering/development?
- Imagine you are given a budget for grocery shopping. SR&ED does not compensate for your cost of purchase
- Justify why you had to grow your own tomato instead of buying existing ones
- Explain why the process of growing this tomato is unknown to you
- Capture the research effort, cost and time
40. TODOs
40.1. TODO https://www.gitbook.com/
40.3. PHP catch fatal error then close MySQL transaction
40.6. dialpad.com, aloware.com
40.7. ClickHouse.tech
40.8. DONE MySQL sql mode
40.9. TODO Vagrant on Hyper-V
40.10. TODO CSS Nesting
40.11. TODO Inspectlet.com
40.12. TODO QUnit
40.13. TODO Learn Continuous Integration "Pantheon Build Drupal with Composer on Travis"
40.14. TODO Learn tool Zeplin
40.15. TODO Lynda Web Project Workflows with Gulp.js, Git, and Browserify
40.16. DONE Lynda Agile Project Management
40.17. TODO Lynda Firebase, Travis CI, Heroku
40.18. TODO JavaScript HTML5 Animation Framework: GSAP or Tween https://greensock.com/
40.19. TODO Lynda: MVC Frameworks for Building PHP Web Applications with Drew Falkman
40.20. TODO Book: Code Complete (2nd Edition, Microsoft Press)
40.21. TODO custom-elements-everywhere.com, williams sonoma
40.22. TODO gridbyexample.com
40.23. TODO BrowserStack and BeHat
40.24. TODO SearchKings.ca
40.25. TODO Agency DriveDigital.ca
40.26. TODO Front End Design
40.26.1. https://codekitapp.com/
40.26.2. https://webpack.js.org/
40.26.3. http://postcss.org/
40.27. TODO Web agencies backed by WP Engine
40.28. TODO https://developers.google.com/training/
40.29. TODO https://grow.google/
40.30. TODO https://www.thinkwithgoogle.com/intl/en-ca/
40.31. TODO https://experiments.withgoogle.com/chrome
40.33. TODO https://generalassemb.ly/ Course on Product Management
40.34. TODO https://www.roberthalf.com/ Job Hunt
40.36. TODO http://grow.io/
40.38. TODO Learn New Relic One
40.39. TODO Learn https://langserver.org
40.40. TODO Python Standard Library Essential Training
40.41. TODO Learn Rust to develop API, Restful Microservices
40.42. TODO OMCP certification
40.43. TODO Certified ScrumMaster (CSM) vs Professional Scrum Master (PSM)
40.44. TODO Book: Nonviolent Communication: A Language of Life
40.45. DONE Lynda Behance.net
40.46. DONE Lynda Illustrator CC for Web Design: Core Concepts, Aesthetics, Image Optimization, Wireframing, SVG
40.47. TODO Semantic-UI.com, Slick JS, Typed.js
41. Random
- People
- How many web developers? Are they responsible for all websites and enewsletters?
- In-house Ad Ops?
- Who will I be working with?
- Process
- Who will give me orders and who should I deliver my work to?
- Any milestone I should meet? Busy time in a year
- Product
- websites, enewsletters, any products designed or made specifically for clients? e.g. brochure or website
- What products will I be responsible for?
- Do you have some goals that you want to achieve for short term and long term?
- Key Intelligence
- Is it a system?
- What do you expect from me?
- Melissa Young Sing
- e-commerce website. Improve user experience
- A role between Product Manager and Development Team. Documentation
- Build an AdOps team
- JS: object vs array
- js implicit parameter